Transcription Hello my name is Paul FLOYD and I’m going to talk about “Upstreaming the Valgrind port to FreeBSD” Slide 1 Title OK, Upstreaming the Valgrind Port Stlide 2 Agenda Today I’m going to talk a little bit about myself just to say how I got to be working on Valgrind on FreeBSD, I’ll give a bit of an introduction to FreeBSD and to Valgrind for those of you that are not familiar with them. Then the bulk of the talk will be about the tools and the way I’ve been working on Valgrind and also a selection of the big issues that were required to be fixed. I’ll finish with a bit of a conclusion, future work and a few other little remarks. Slitde 3 About me So a bit about me Slide 4 Map arrow Tondu Was born and I grew up here in a village called Tondu ( IPA Tondi: ) in Wales. After that … Slide 5 Map arrow Manchester … I went to study at university here in Manchester in England where I studied … Slide 6 Manchester Uni … Electronics. A bit of a long story how I got from Electronics to software development but it can be done. It was also in Manchester that I first installed FreeBSD. That would have been late 1995 FreeBSD 2,10 32bit on a Dell 486 PC with 16Megabytes of RAM. Also in Manchester I met my future wife, a French lady so after a little bit I loved to France. Slide 7 I’m not going to talk about all my career so I’m going to skip 20 years and … Slide 8 Map France … for all that time I’ve been in France, here, in the South East in Grenoble next to the Alps.Grenoble’s a really nice place, I like it here so a bit of advertising … Slide 9 Photo Chartreuse … here are a couple of pictures I took on my walk to work. This is the Dent de Crolles Slide 10 Photo Belledonne And this is another picture taken from the same place looking South to the Belledonne chain of mountains which is really the edge of the Alps. Slide 11 Siemens I work for Siemens EDA, that’s Electronic Design Automation. We make software for all aspects of electronic design. In particular the team where I work produces a toolbar analogue simulation and my focus is on performance. At work it’s Linux our main platform and I do use Valgrind quite a bit but I don’t use FreeBSD at work, only in my spare time. OK that’s enough about me. Slide 12 About FreeBSD Now a few things about FreeBSD. Slide 13 FreeBSD version support At any one time will be three versions being supported. CURRENT is the development branch, at the moment it is 14.0. STABLE is when CURRENT has become stable enough to start preparing for a release. Usually it takes about 9 months to go from CURRENT to STABLE. There are always 2 STABLE branches, at the moment they are 12.3 and 13.0. When a release is made it will be tagged and taken from the STABLE branch. The STABLE branches will be supported for 5 years. From a Valgrind perspective it’s the CURRENT branch that’ll be causing all the problems. This is where unstable changes will happen, might change ABI or use a new version of clang of something like that. It’s always good to track the CURRENT branch to stay ahead of any breaking changes. Slide 14 Platforms What platforms does FreeBSD run on? AMD64, i386, PowerPC, ARM and RISC-V Just a quick word about i386. That’s the name that’s used in the FreeBSD world with is a but different from the x86 name that you normally used in the Valgrind world and in Linux. For historic reasons. A couple of old plafforms are being phased out MIPS and SPARC. Of the first set of platforms AMD64 and ARM aarch64 only are tier 1, all the others are Tier 2. I386 has been deprecated on Linux for quite a while but only last year was it dropped from Tier 2 to Tier 1. I think that is is for historic reasonls, a lot of people have a sweet spot for i386. The very first open source version of BSD was actually called 386BSD. Slide 15 core and ports How the software is organised in FreeBSD is perhaps a bit different to a typical Linux distribution. There is what is called the core which is not just the kernel but also a minimal user land, there’s the shell and also compiler, linker, vi and tools like grep and sed. Anything that is not in core will be in a port. So, these are either pre-built binaries or you can build them from source and install them yourself. Each port has a maintainer and for Valgrind I’m the FreeBSD maintainer. Maintainers can’t actually make changes to the FreeBSD git repos, that can only be done by Committers. A maintainer has to do some work and then ask a Committer to make the change. Slide 16 ports So here I’ve got a couple of pictures. Here is a screenshot from the actual Makefile from the Valgrind port on FreeBSD. It’s fairly simple, really just a wrapper around the Valgrind build system, also saying the version number and where to get the tarball. Frechports is a fairly useful web site which you can use to browse and look up information on FreeBSD ports. Slide 17 other web sites So, speaking as a FreeBSD maintainer there are three web sites that you need to know and have accounts with. The FreeBSD bugzilla site. You might get some bug reports for your port here or even using your port but it’s also the mechanism used for uploading patches to the port. Like I said the Maintainer can’r commit, you need to create a bugzilla item, a Committer will check it and actually make the commit. Two other web sites to know of. Phabricator for discussion and approval of patches mainly in the core part of FreeBSD and also there’s a FreeBSD wiki with lots of general documentation. There’s a page there for the Valgrind port. Slide 18 differences wrt Linux I’d like to talk about a few of the differences between FreeBSD and Linux, in particular with an emphasis on Valgrind. So first of all the user land tools like sed and awk and grep are different, They are mostly compatible but not entirely. Most of the time this doesn’t cause much of a problem but for the configure script generated by automake there can be differences. On FreeBSD you can install the GNU versions of these tools but usually they have a different name. So there would be sed and gsed. The second item here, the compiler and linker. Not the same as the default on Linux. In terms of inputs GCC and BFD ld the defaults, they’ll take the same command line options and compile the same source pretty much as well as clang. From a Valgrind perspective no big problems there. The differences where they really start to count are on the outputs. The code generated by clang and also the ELF segments produced by lld can be different. And both of those can be big problems for Valgrind. FreeBSD has its own libc. It doesn’t use GNU libc. Mostly this doesn’t make any difference but just occasionally it does. Just to give a quick example of that the semaphore functions are in libpthread on Linux but they are in libc on FreeBSD. There’s no /proc filesystem by default on FreeBSD. It’s an optional installation. That doesn’t have much difference but it does change some things like the interface used when looking for filenames. And lastly different, different filesystems. This doesn’t have much impact on Valgrind but it does occasionally change things like the size of buffers which will affect messages for things like write system calls if there is an error in the arguments. That’s my introduction to FreeBSD from a Valgrind perspective. Slide 19 about Valgrind A bit of coverage of Valgrind for anyone not familiar with it. Valgrind is a fairly small world, there are about a dozen or so active developers. sourceware.org is the host. It runs on Linux, FreeBSD Solaris and macOS (not very well on macOS at the moment). And Linux supports quite a few different platforms AMD64 x86 MIPS POWER ARM and also IBM s/390. FreeBSD only runs on x86 and AMD64 at the moment. Slide 20 tool startup I want to talk a bit about the startup process for a Valgrind tool. This is all very low level and essential for getting Valgrind running. So the first thing that I want to say when I describe the startup process is that fundamentally Valgrind is a single process running. Some people tend to think that Valgrind is a bit like GDB using PTRACE with 2 processes but this is not the case, there is one process. It’s the Valgrind tool with the tool text code in memory, its own heap and stack and then separate from that there’ll be the guest program that is being tested with its text in memory as well as its heap and stack. OK, so startup. Valgrind, the tool is entirely standalone and does not link with libc. This means that there is no ‘start’ function to call main so we have to have our own start function. We create our own stack and calla function called start_in_C_freebsd. Again, so far these are all just small functions just doing things like setting up the stack. This then calls Valgrind main. This is the common function for all platforms. This function is about 380 lines of text. It’s long and it does a lot of stuff, and all very important. Quite early on it has to do some command line argument processing and then it starts debug logging. So this means that if you do have any problems in this very early part of the code you’ve got no logging and debugging means using GDB down to the machine code level. Slide 21 tool startup 2 OK, second slide here. Then the tool will start the address space manager and dynamic memory. It finishes command line processing and loads the guest image. This is quite an important and complicated bit of code. Valgrind needs to load 2 special shared libraries, one for the core and one for the actual tool (one for each type of tool that Valgrind can run: memcheck DRD and so on). And in these libraries there are some special functions with mangled names which are used for things like redirecting the guest functions. Then we set up the client stack and this will also set up the command line arguments for the client the environment variables and the auxiliary vector. So the auxiliary vector if you are not familiar with it, it is an array of values , about 20-25 of them containing things like the number of CPUs the operating system version, stack protection flags and things like that. Then if it’s available we read the debug info. And finally we start executing the guest. Again I’m somewhat mixing things up. In Valgrind we often speak of the client and the guest. It’s the same thing, Slide 22 basic guest execution Just a few words about how the guest runs in the Valgrind tool. It's all running in VEX. So VEX is the virtual machine. It's like a superset of all the CPUs that Valgrind supports. It executes as a JITter (just in time), so by default it will read 50 instructions, convert them into VEX opcodes and execute them. In Valgrind signals are all handled by the host. Syscalls and ioctls are also intercepted. 2 reasons for that. One is that the arguments need to be checked, so we need to check that it's not pointing to invalid memory. But also because some sys calls need to be handled by Valgrind, things like process management, thread management and things like that. One thing that I forgot to mention there, for FreeBSD the code in VEX required almost no changes. There was one thing that I needed to do and that was related to one type of error that causes a sig segfault on Linux but causes a bus error on FreeBSD. And last of all each tool may have its own intercepts for various functions. Mtemcheck intercepts all of the malloc family, the thread checking tools DRD and Helgrind, they intercept the thread functions. Slide 23 Lines of code I've got a couple of slides showing how much code there is in Valgrind and how it is split amongst the various directories. This is with the regression tests. And the obvious thing here is that VEX, the virtual machine, is the biggest. The 'none' directory is very big as well, coregrind pretty big then memcheck and all the rest are pretty small. Slide 24 Lines of code again Here's the same pie chart, this time I've taken out the amount of code in the regression tests. This is good news! VEX is quite a bit over half the source code of Valgrind and as I said earlier I almost needed to make no changes in here at all. Almost all of the work that I did was in coregrind. This is good! I must congratulate the other Valgrind developers for writing the tools and VEX in a way that is platform independent. At least, as long as your CPU is supported. Which has made my life much, much easier. I forgot to say there ... Back to Slide 23 ... about 3/4 of a million lines of code in total. That's in a clean git repo. When you actually do the build of Valgrind it generates some code and will go up to about 950 thousand lines of code. Slide 24 again Without the regression tests about half a million lines of code. Slide 25 About Valgrind on FreeBSD OK that's the introduction over. Now I'll talk about all the things regarding the port and the work that I've done. Slide 26 history So first of all a bit of history. It's been a long long story. Initially the Valgrind port was done by Doug Rabson. And the first trace of I have of that in the git logs is back in May 2004. Stan Sedov took over maintenance back in 2009. There have been many many contributors over the years. 20 or 30 I guess. And I started working on it properly January 2020. I became maintainer of the port in April 2021. Jumping the gun a bit here the FreeBSD code was merged into the sourceware.org repo back in 2021 in October. That was part of the release operation for Valgrind 3.18.1. Slide 27 port status So when I started working on it what was the status? Well it was broken. It hadn't been very well maintained for a few years. There had been a change to lld which added a new read-only non-fixed ELF segment that Valgrind didn't recognise and that was causing the guest loading to fail. On 32 bit platforms, totally broken just about. Even without those two the thread handling code was unusable. Also there were no regression tests for FreeBSD. Not a very good sign for quality. Slide 28 dev env A little bit about my development environment. I've got a fairly old HP workstation running FreeBSD 13.0-RELEASE. I don't have 6 computers so I use VirtualBox for all the other platforms, 32bit and the unstable branch and the older branch. My PC is triple boot so that means that I can boot Linux and if I think that something is a bit dodgy that I've done I can check that I haven't broken things on Linux before I push it. Now I'm going to talk a little bit about the tools that I've used on FreeBSD. The first one ... Slide 29 dev tools ktrace ... a big part for any new platform is handling the syscalls. So ktrace is a tool like strace on Linux. It's very easy to use, you just type ktrace, you can optionally add -d if you want to trace descendants and then the application and the arguments. IOt's very good for getting a quick output of the syscalls. This example here is showing a small standalone executable outside of Valgrind. It just has 'main()' and a 'sleep()' call. So quite often what I need to is to compare this output with the same application running under Valgrind. If there is a problem with Valgrind's wrapping of a syscall and it's not making the call properly this will be a good indication. I can also compare this to the output of trace on Linux. Slide 30 dev tools dtrace Another similar tool is DTrace, this comes from OpenSolaris. It's more powerful in that you can write scripts. So the example here is a script that I wrote debugging a problem with the sysctl syscall. I copied and pasted the sysctl interface from the manage, here is the kernel interface, here are the interfaces to match and some code to retrieve the argumetnts and print the 'name ' or 'OID' of the sysctl. Slide 31 dev tools binary analysis When you are working on any platform, I think, with Valgrind you have to work with other binary analysis tools. Elfdump and objdump. You need to look at the ELF segments. As I mentioned earlier on sometimes lld will change things and to debug that sort of problem you need to look at where exactly things are in the file with these tools and compare it with what you are seeing in Valgrind. Very very often you need to dump the assembly code for the guest application just to see what is going on. A typical problem is Valgrind will run and you won't see any error. The question then is, is this a problem in Valgrind failing to detect the error or is simply that clang has optimised away the code. That happens quite often. Slide 32 Valgrind traces Not really a separate tool but very very useful is the traces that Valgrind can produce. It has many options and lots of the code is instrumented for tracing in Valgrind. The first one is minus v for verbose output which you can repeat several times for more output. And similar there is debug output, and again you can repeat to get more output. Lastly there are --trace options which which do tracing which is more targeted to a single problem like syscalls redirections and signals. There's only one real problem with this, is that is can be extremely verbose. It's quite easy to get log files that will be 5, 10, 20 100Megabytes in size. Even when you are working with a very small test application. So this can make using things like diff a bit difficult. Slide 33 Valgrind address space This is the output from Valgrind with -d -d. It will print, the address space manager will print I think 3 times during execution the memory map of all the Valgrind process itself. So here you have a little header showing files, the 'sap' application, the link loader, the preloads and so on. I had to edit this to make it smaller to fit on the screen. Here's the 1,73, that's the text of the guest program. Here we have the shade libraries. Down at the bottom is the stack, the client stack and the host stack. It's all fairly clear and with practice you can read it quite well. Slide 34 procstat memory map Here's the same thing with the FreeBSD tool procstat, procstat -v. Again. It looks very similar to the previous one and if there are problems with allocating or loading things into memory it's very useful to compare the two. Also on Linux you can have a very similar output for pmap as well, pmap -x. Slide 35 freebsd source Not really a tool. Access to the FreeBSD source. The kernel, libc and the thread library libthr, not sure how to pronounce that. So I have had need to compile a custom kernel and I've also needed to build both debug versions of libc and libthr, as well as just reading the source code. It's all very useful. And the last thing here is a file called syscalls.master, a text file describing all the syscalls. So this is something that I've used very very often, I've bookmarked it in Mozilla, very very useful when adding new syscalls and just making sure that the interface is correct. Slide 36 fixing the port So, OK, the actual work that I did on the port. The first thing that I did was clone the official source ware git repo. A few other people had already done work in trying to fix things, So there were 2 other repos being worked on by Phil Longstaff for the first one and Ed Maste and Mark Johnston for the other one. I merged all their code into my local copy of the official repo. Things were pretty rough at the beginning so I started working using GCC rather than clang and a purely debug build. By default if you build Valgrind it'll build with -g -O3, a mixture of debug and optimised. That works pretty well but when you're in GDB, the debugger, quite often you will see that variables have been optimised away and it's easier to work with pure debug code. One of the first things that I did was a load of code cleaning. I don't like compiler warnings, I always like to have zero warnings so I tidied up all that code. I added version checking for the version of FreeBSD along the same lines as Darwin. In retrospect I'm not really sure that it was necessary. Well, it's there now, I'll keep it. Then I fixed a couple of really blocking issues, there was a problem creating the stack on 32bit and as I mentioned earlier this problem with the non-fixed read-only ELF segment. Shortly after that I created a GitHub repo for this which I've used since then. Slide 37 reg tests Amongst all the problems several fairly frequent, recurring problems were arising with the Valgrind regression tests. A lot of our tests contain Undefined Behaviour. For instance they might do something like a pthread_create without calling pthread_init. The result of that (big shrug) varies from platform to platform. On Linux it might work with an error but it might hang or crash on FreeBSD. So for some of these I had to make changes to the regression test and sometimes I had to add a new reference file, what we call the expected, for that test. Clang optimisation. So I've already mentioned that clang would optimise things away. Usually the fix for that would be to add, to make a variable volatile or to mark a function as attribute noinline. The third quite frequent problem is with the code generation and in particular with the ternary operator clang will emit a "cmove" conditional move opcode while GCC will produce a compare and a conditional jump. The difference here is that the cmov doesn't immediately cause an error in memcheck. But when there is a compare and conditional jump there will be an error at the moment of the conditional jump. So this can change the actual moment that will report errors. So now I'm going to talk about a few of the bigger problems that I fixed, ones that I think are fairly interesting., tended to be on x86. Slide 38 auxv pagesize 1 The first problem was that there was a failure at the very beginning of guest startup on x86. As I mentioned, when the Valgrind host synthesises the client stack it also creates a copy of the auxiliary vector. It's not an exact copy. One of the things that Valgrind does is strictly separate the guest and the host memory. And some things in the auxiliary vector are pointers to strings of things like that. And as a rule Valgrind doesn't synthesise those. So one of these things is a pointer to the PAGESIZES array. This is the hardware pagesizes that the hardware supports. So normally the link order needs to know about these and it will just read the values from the PAGESIZES from the aux vector. But Valgrind doesn't synthesise one so it (link loader) sees that it is not there and it will use 2 sysctls to try and get the pagesizes. One to get the size of the array and the second one to actually read the values. Slide 39 auxv pagesize 2 On a pure 32bit system it was OK, on a pure 64bit system it was OK but when it was 32bit on 64 there was a problem. In that case the sysctl should return a length of 3 which is the right value for a 64bit system but it should be returning 2 the length on a 32bit system. When the second sysctl call was made to read the vector of sizes the kernel was seeing a request for 3 elements but it was only expecting a request for 2 And that caused the sysctl to return ENOMEM Oh dear! So the link loader was failing and that was the end of the guest. Nothing really happened. I fixed this in Valgrind with a really ugly bodge, I actually did add a synthesis and allocation for the PAGESIZES in the auxv. This would cause some error messages in memcheck so I had to add a suppression. Very ugly, but I had no choice. It was that or nothing. The good news is that I reported the bug upstream in FreeBSD and it was fixed back in May 2020. So the newer versions of Valgrind no longer need this hack. Slide 40 ld_preload The second big problem, once I manages to get x86 to actually run it wasn't detection any errors with memcheck. OK, so during the startup the tool will set LD_PRELOAD so that the guest will have will have the core and tool shared libraries loaded when it (the guest) loads. On a pure 64bit system and on a pure 32bit system the environment variable for that is LD_PRELOAD. However for a 32bit application on a 64bit kernel there is a different environment variable called LD_32_PRELOAD. And this wasn't even being set by Valgrind so it couldn't possibly work. In addition there was a function that was supposed to detect 32bit on 64 and that wasn't working either. It was using the hardware/machine sysctl that was working too well and it was saying it was 32bit on 32bit. My fix for this was to look to see if the link loader had the name different from the usual pure 32bit elf32.so. On a pure 32bit system this will be just elf.so. And I also set correctly LD_32_PRELOAD. Slide 41 debug printing Third problem with x86. Whenever I used the -d for debug printing it would crash. As I explained earlier we want to have the debug printing available very early on so that means that Valgrind has a very low level function to, for debug output. It's inline assembler making a write syscall to stderr. On i386 FreeBSD takes all its (syscall) arguments on the stack. But it looked to me as though the code was just a copy and paste from Linux and it was passing arguments in registers. As you can imagine when you do a syscall and it's looking for something on the stack that's not there it's going to crash, and it did. The fix for this was fairly straightforward, it was to rewrite the code to use the stack. It is possible to pass arguments in registers but that requires setting a bit in the ELF headers and that's not something that we want to do. Slide 42 DRD threadid OK, my last bug, I think, is a bug with DRD the thread checking tool and I was getting an assert in applications loading a shared library. DRD needs to track the threadid for each thread, used in error messages for instance. For the main thread it tries to read the threadid very very early in process excution. On FreeBSD this is before libthr has been loaded. On FreeBSD libc contains stub functions for the pthread functions. So this is the one that was being called, the stub function. And that means that DRD was getting a junk value for the threadid. A bit later on in the code there was an assert checking that the threadid was something sensible and obviously the junk value didn't match and it would assert and abort. So my fix for this was to do a dlopen and straight away a dlclose of libthr and this would keep it in memory so that the call to pthread_self would work correctly. Slide 43 sigreturn 1 OK so this is a bug, this is one that I haven't fixed yet. It's probably the one that took me the longest to understand. I'm not sure when or if I'll be able to fix it. The problem is that for code that uses signal handling we get a kernel message complaining about something. This diagram here is showing the flow for an application running outside of Valgrind when there's a signal handler. So the user code is running, a signal happens, perhaps the user presses ctrl-C, the kernel signal handler runs there's like a context switch into the (user) signal handler, that returns to a little bit of assembler, something called the trampoline which does a sigreturn so it does like another context switch back to the user code. So that's the normal flow. When we're running under Valgrind it's the same thing except the user code and signal handler are running under VEX, it's Valgrind that's doing the signal handling and we also replace the trampoline and sigreturn. This is also the case for Linux and FreeBSD without libpthread. Slide 44 sigreturn 2 When FreeBSD links with threads the picture changes a little bit and there's an extra function that gets inserted. thr_sighandler. So the flow is that it goes from the (kernel) signal handler to this thr_sighandler, call the client (sighandler) code, comes back and it (thr_sighandler) calls sigreturn. And that doesn't work very well with Valgrind I'm afraid. Slide 45 sigreturn 3 This is the picture when it's running under Valgrind. The user code runs under VEX, the signal happens, Valgrind handles the signal, unfortunately we don't know that it's calling this thr_sighandler, again this is all under VEX. We come here (user sighandler). The thr_sighandler want to call sigreturn but we don't want that and it fails. Thankfully! And it produces a message like this (sigreturn flags = 0x4) Somewhat by accident thr_sighandler returns and it gets back to our trampoline which allows us to return (to user code) and it works again. IO don't really have an easy fox for this. I think to really fix it I would haver to replace all of the 'signal()' code in Valgrind. We might have to live with this problem. It's not a fatal error. For applications that only do a little bit of signal handling it's not too bad but if you have a program with a rapid timer it will mean that tons and tons of messages get printed on the screen. Slide 46 stats OK I'm coming to the end. A few statistics on the work that I've done. I made about 700 pushes to the GitHub repo. That's also a bit of a reflection on the way that I work, I push early, I push often. I fixed about 145 GitHub issues, that includes like work tracking and enhancements as well as bug fixes. The total amount of work I would estimate about 6 person months all-told. About 9000 lines of code added in coregrind. And 45 new tests. Slide 47 future work OK, future work. I'm afraid with Valgrind your work is never done. There are a few missing features. There's no vgdb invoker so that means that if you are running gdb to attach to Valgrind if you hit ctrl-c it won't work properly. There's a bug in thread local storage in Helgrind that causes false positives. That's going to need some fairly low level work tracking libc creating thread local storage. Not too sure how possible that is. There was a horrible bodge that I did for handling a change to lld in version 9 regarding the readp-write PT_LOAD being spit into two parts. And OMP generates loads of errors on FreeBSD with llvm. In addition to those big things there are quite a few small issues. Mainly cosmetic and cleanup. So that's the situation now and a small list of the things that I'd like to do. Slide 48 so what? So the conclusion, who is using Valgrind, what is the result of all this? I can't say anything about the download or usage, I haven't got any information. But I have had a few bug reports. The rust maintainer on FreeBSD reported that rust (applications) was crashing on startup because of a stack problem. This was partly because rust was creating a user stack different from a C application. This problem has been fixed in rust and also in Valgrind. So this (perftools) wasn't really a Valgrind problem. The google perftools used by Prolog on FreeBSD, normally it detects Valgrind and it doesn't use its own memory manager which conflicts with Valgrind. So I got a fix done in google perftools and that now works OK with SWI Prolog. I had a report of a crash from the glusertfs maintainer. There was a bug in my code handling the 'swap context' syscall. And the last one that I had recently was from the HardenedBSD team. First of all this is like a security hardened spin of FreeBSD. But they don't use the FreeBSD 10 compatibility interface and there just happened to be one call to a function like that in Valgrind which was easy to replace. And the application that they were working on was using a new syscall called realpathat which I hadn't implemented in Valgrind and that's done. Slide 49 thanks OK, last slide. It's been a long and windy road but I haven't been all on my own. I'd like to thank all those people that have worked on the port before me. I'd like to thank the people on the FreeBSD hackers and toolchain mailing lists I'd like to thank my fellow Valgrind developers for putting up with me And last Nick Briggs who helped with the 32bit port, 32bit version. Thanks a lot Nick. OK that's the end, thank you for listening and goodbye. Slide 50 (not in video) Contact details