A road named NoNameUno

Lately I've been  hacking quite a few open source code in the telecom and data com area. One of the interesting thing I came up ( and of course nothing new) is the software organization. I've read Hardware Organization books, but I don't recall any such book on software that I read. There are a few books on crafting good codes with our choice of languages... But what about the organizations. Even source code could make anyone confused if they are not logically organized.

In this particular case I'm talking about some user land libraries. When it comes to closed source, all one needs to do is to expose the contract very very clearly and precisely, and rest of it is hidden anyway. For open source, there is really more responsibility. Without looking at the implementation(s), some of the things I would really like to see are --

1. Clear separation of interface files, so an user knows up front what are the interfaces.
2. Reflect the naming of files and the organization of files in a logical manner that mimics the specification, for example the functional design of an RFC from ITU.
3. Immune to refactoring meaning no illogical introduction of files and or the tree structure.
4. Examples of isolation of interface, so that the bindings are very very loosely coupled.
5. Replacing old libraries with new one should very very seamless.

Few such libraries seem to have some respect for those five points I just stated, but many don't....

What is the value of :  2 + 2 + 2  ?

Using similar operator(s)

1   1   1    = 6

2   2   2    =6

3   3   3   = 6

4  4   4    = 6

5  5   5    = 6

6   6   6   = 6

7   7   7   = 6

8  8   8  = 6

9  9   9  =  6

Memory, who does not want more? How about quick fetchers!. Remember, the quicker we answers qizz, the smarter we are. Well, at least that is the perception of lot of people...

In this case, I needed a bunch of quick memory ( yes that is pinned memory, and they are not paged out). Why?. Because, I was dealing with a near million lines of code for a project, with 15 or so processes, including NT services. They are mostly with PSTN related call processing and retries and timeout could be very bad for user experiences. Putting anything into the kernel mode was the least priority. But eventually few components will deserve to have sits in the "Senate or perlament of kernel modes". Fewer the better...

If I want to get my hands dirty, sure I can use KM code to have non-paged allocations and hand them over to UM. But I was looking for something in the user space. The nature or the pattern of the allocation is that the processes would acquire dynamic memory right around the start. Then they would return and again would try to acquire dynamic memory. Lot of those small allocation don't necessarily get freed but a good percentage gets freed and later allocated again...

First job was to findout, where most of the allocations are happening. Basically finding out the route of code paths that are doing the memory managment work using system heap. Luckily, there was one dll that was doing the job in majority of the cases, and is being mapped to all the processes. Looks like I found the Gold Mine !.

Now to sit back think about, how I could have the base ( often called slabs) allocation from non page area without using the kernel mode coding... Come to rescue is the AWE method of Windows. So now I know, I can ask for pinned memory from users space, well of course, with little dances. And I can map those allocation to different usermode address spaces.

Now it is the duty to find what would be my least foot-print in this change. Remember, this is near million lines of code, and mostly open source and some licensed. Rest is ours own design. As always, some form of allocators ( be it malloc or calloc or new or ...) are used, and we need to inject something to fool the preprocessor into going to our heap manager instead of Windows runtime. So that was easy, just defines those std allocator routine names with our heap-manager-allocator name and keep it in a header file. The header file would be included depending on the build type...

Finally the turn is to comeup with a heap manager on top of the slab allocator using AWE.  As it turn out, it is at least six month work to get a decent allocator, though who are experts in this area might turn out something at a shorter time frame. So I went on to look at open source allocator(s), as my boss is from open source area. There we found a few, but DL allocator seems to be the best. Why?. Because the site says it is being used by lot of products, and some variation of it is also used in some distro of Linux. This might give a false sense of solidity of the allocator, but for a fairly small shop due deligence also helps sometime.

Observation ---

The product runs over 7 days with fairly high stress, and observing the delta page fault and total page faults using sysinternals procexp.exe utility shows significant improvements...

Well, when your domain of debugging discourse is quite large, about 14 or 15 interacting process in a system talking to other systems with same number of processes, debugging could be real fun!. One would feel like, when would it end?. In these cases analyzing traces are one great method to find out who is doing what. So there has to be a distributed logging mechanism, that one can find one just by searching the internet ...

Once we have the logs of GB sizes, it's not fun to look at them using notepad, so need some kind of intellegent tools to filter out as much as unnessary information as possible.... And here the command line utilities of Windows are bit behind than using Perl or other unix tools that have been ported to Windows...

I used Perl to filter out the logs: By process; By threads; By column, By regular expression. Then redirect them to other files to sort even by fields of interests to have lot of informations that quickly converge to make a decision about what the bugs are...

Wish I could try tearing apart such a log, and then merge it again using inverse process. So one would be to skin out a column then merge it back and things of that nature ...

About three years ago, a relative of mine with physics background was talking about transmitting electrical power over wireless medium. Honestly I could not quite comprehend the problem, but at least I knew this fellow did not know how it could be possible. And what are the fundametal barriers was not know know to him either.

But just to let you know ( in case you missed it ) that MIT already had a prototype about transmitting power over wireless. There might be some bad effect due to radiation and what not, but the prediction is that in near future we don't have to use the good old rope technology to power our gadgets.

I would surely like to see everything is wireless. I simply hate to connect computers with other associated gadgets ( including KVM and network switches ). This rope technology is very detasteful for me.