future of standardized tsection.dat

[timbooth]

I have concerns about the future of standardised tsection.dat, and thought I'd raise the issue here so that it can be discussed.

As many will know David Beach, the current 'manager' of the standardized tsection.dat scheme, is standing down and seeking a replacement. Whilst an equally keen replacement may be found, I feel that a better system would benefit all users and reduce the management of standardising track/road systems. There are several issues with the current system that are only going to worsen as more track/road systems are developed.

1) admin overhead
At the moment, developers send a list of shape entries to David and he then enters them into his system and produces a tsection.dat for developers to test, before issuing the official build release. The problem with this is that David needs to keep issuing a new test build each time a new set of shapes arrive - which I can imagine is very tedius - or leave those shapes until the next release. Also, its a bit of a duplication of effort for developers to set up the tsection.dat entries, which David then records and then puts into to the tsection.dat build. Also, each time an error is detected in testing, or new shapes arrive, a new build has to be compiled and tested - which affects all developers involved, who have to repeat their testing again.

2) Release interval
The above issue, and the impact of a large number of new entries, means release intervals are very infrequent. Its not possible for developers to issue new shapes as and when they would like. As a developer, I would like to make new shapes available as soon as I have built and tested them, but this is not possible at the moment. For instance, to issue just a couple of new shapes to get around a problem isn't practical.

3) Build version
When a route needs to be installed, it has to have a minimum build of tsection.dat build or else it will fail. Its easy enough to install a tsection.dat of the right build at installation, but less easy to prevent it overwriting a new version - leading to routes requiring a new version failing. A commercial route on the shelf in a shop could be several months old, so the tsection.dat will be out of date when installed. Not all users will know if the tsection.dat has been overwritten by an older version, leading to all sorts of errors when running MSTS.

4) installation
The main problem with tsection.dat is that each time you install a new build, the list of shapes in RE gets longer and longer. All track/road system shapes have to be listed, but users that don't have them installed have a lot of shapes that they don't need. Whilst tools like RouteRiter can strip out uninstalled shapes, this has to be done after each tsection.dat install. Users find it difficult to know what shapes they can and can't use, unless they happen to know what shapes they have installed. Also, installation of new shapes is not straightforward, as the tsection.dat file needs to be updated separately.

I therefore think a simpler system would make sense that still retains compatibility, but avoids the issues listed above. I've been giving this some thought and have come up with the following simple scheme.

When developers make new track shapes, they also write a small config shape for each new file - eg. shape.dat - which contains the tsection.dat entries for just that shape. These dat files are included with the installation instead of a tsection.dat file. Instead, a simple utility is run after installation which reads in all the .dat files and compiles the tsection.dat, and can then launch MSTS if necessary. The same utility could be run and used to launch MSTS each time, therefore ensuring tsection.dat is always up to date.

The utility would also be used by developers to verify that there are no errors in new entries. This means there is no real need to verify each entry as is currently done by David. So long as developers test new shapes thoroughly, using beta testers, then no serious harm can be done - and any errors that do slip through can be fixed quickly and easily by the developer.

This would dramatically improve the installation of new routes, especially commercial ones, as there is no longer any dependency on build version - the route and track shapes can be installed together, and the utility will update the users tsection.dat if necessary. Only shapes installed are available in the tsection.dat, so users have confidence that they can use all shapes in RE.

The management of track shape/path ID's would have to continue as before, but thats all the manager would have to do - meaning anyone could take on the role easily. The allocation and admin of ID's could be moved online to make them easier to manage and track.

As a developer, the current system does complicate the issuing of new shapes and means I can't quickly issue new shapes that people desperately need.

There is also a real problem if David fails to find a replacement before he departs, or if the replacement gets overwhelmed by the growing number of track shapes. Without some kind of system in place, the future of UKFS and other track systems will be in doubt.

[anamorph]

Well, that sounds a perfect solution, only - as you say - will it work out that way.
Nothing says the tsection file has to come from one source, and as long as everyone (as they do) has the standard, thay only need to update
for any extra system they choose to use.
Having one author for a file of this type only delays things in the long run regardless of the updates needed - how long to wait to add 10 more updates etc etc.

Hopefully something along these lines can be sorted out - I can't see it
causing any problems, and certainly very little hassle for end users.
And of course, life much easier for track system builders...

[saddletank]

What response to this proposal over the pond, Tim?

My view has always been that the robustness of the standardised TSection system is that it is well controlled and the chanelling through 1 person maintained order and continuity. We have seen what a shambles was made of it by some commercial teams who eschewed standardised TSection and did their own thing - result screwed up users MSTS installs (Maglev and that Dutch route of last year come to mind).

[timbooth]

I've sent an email to David to get his comments.

I'm not seeking to undermine the standardisation part of his project, that should continue as before as its vital. We still need the control of shape and section ID's, and routine housekeeping that David has been doing.

Yes, there are advantages of David auditing all new track shapes and controlling the tsection.dat release, I just feel its not the most efficient way to do things - and thought a discussion might help improve the system for all. I know David doesn't look forward to my submissions of 400+ shapes. The work involved can only increase as time goes by.

Also, what happens when the standardized tsection.dat is no longer updated (either because of a move to support MSTS2, or project closure) ? I'd still like to develop UKFS for MSTS1 while there is still demand, but if the tsection.dat is not being updated that would not really be possible.

[NeutronIC]

I guess a util like Train Store or my Chooser could effectively then switch out the different track types so you choose the route, it loads the trains/routes and updates the tsection appropriately by basically just reading a config file in the route directory that lists all the shapes etc to move in to global, all the entries to add to a default tsection etc - with a back up of the default, it would always bring the system back to being a default tsection then upgrade it. Plus you'd have a lot less in memory at once - so if you are using an XTracks route, no UKFS in memory and so forth. A more advanced version could look at what track is actually used in a route and only load just those bits in, more efficiencies to be gained.

With the whole thing being centrally managed that would offer some advantages, from my perspective, if it can make it easier for the average person to install a route without worrying about if installing Paddington/Penzance will knacker their London Brighton for example (because Pad/Pen default install will put an older XTracks than LBE uses if the user doesn't do the install manually).

Maybe this is something that can be largely automated? If I set something up that enabled someone to register a track piece/pieces and then receive back all the shape ID's, it could also then spit out the config files and tsection extracts for each type of track? That would be quite easy to do - would it do the trick ? If so, i'm happy to do it...

Matt.

[saddletank]

Sounds promising. I'd do some testing for you.

[longbow]

Could the Tsection guru not delegate some of the standardisation process by authorising the principal track sytem builders to release their own interim Tsection updates? They could co-ordinate to ensure that each interim update incorporated all previous versions until such time as the next "official" Tsection release.

[TFormoso]

FYI This doesn't address the problem of generating new TSECTION.DAT files, but the next version of Train Store will swap TSECTION.DAT files.

For each Route the user will have the option of specifying a specific set of TSECTION.DAT parameters.

The options will be :-

Any TSECTION.DAT
A Specific TSECTION.DAT
A Specific TSECTION.DAT (or one with a later Build)

(The options can be applied to Simulation Mode and Maintenance Mode, and the settings can be different for each mode)

If no TSECTION.DAT parameters are specified, the settings will default to a system default specification which the user can set.

When unstoring, if there are multiple Routes to be unstored, Train Store will examine the Routes and choose the TSECTION.DAT which is compatible with the Routes. If it is not possible to choose a TSECTION.DAT because of conflicting requirements (e.g. two Routes, each specifying a specific (different) TSECTION.DAT) then the user will be shown a table listing the Routes chosen, highlighting those which are actually causing the conflict, and have the option of removing one or more Routes (from the unstore process) to resolve the conflict.


Tony

[timbooth]

Matt - I was only really suggesting a simple utility (a DOS exe would do) to compile the tsection.dat at runtime (if necessary), or post track install. However, there's no reason why other utilities can't use the same utility for their own purposes. What would be nice is to have a windows application that allows you to enable/disable track/road sets, so that when building a route you can just enable those sets you need for building purpose, but later enable those you need for other routes - or launch MSTS via the application so that it auto-enables those track/road sets actually used. I think this would improve the route builders life.

I think an online ID 'management system' would be worth having, it would allow developers to request new ID's, and verify what blocks of ID's they already have (and whats vacant) and help the manager track ID's and developers. I think individual shape registration may be a good idea as a script could check for errors and clashing ID's, etc. - that would maintain quality control, whilst not requiring anyones time. I think this part is up to David, or his replacement.

Longbow - its the static tsection.dat release that I'm trying to move away from, as its always requires a 'manager' to ensure compatibility and quality control. By using an automated method, the manager only needs to worry about issuing new ID's and housekeeping, and the developers can develop and release shapes at the their own pace.

It will be easier for someone to take on the tsection.dat standardisation project, if they only have a light admin task to perform - but manually processing every single new entry and coordinating testing and releases is a huge task as David knows only too well.

[OkrasaGhia]

Interesting topic indeed!

While I too think that the administrators task should be made easier I don't quite agree with the proposed changes.

Tim's initial posting:

1) Admin overhead
I think the entries sent to the administrator should already be tested. You have all that is needed and can make your own tsection.dat for testing. I always test all entries before sending them to David. If you have done this initial testing properly there should be very little need for changes when you get the test build. If a different developer makes changes to his entries yours are not affected. I think the administrator should only need to worry about possible naming conflicts and that the developer is not using id's outside his range.

2) Release interval
A new build every other month is not that infrequent but I can see your point.

3) Build version
This problem is very real as I have had reports for routes with this problem (LBE installs build #19 and from the postings above I see Pad/Pen has an issue also). Now this would not need to be as a proper installer can solve this problem. The XTracks installer in use since v3.7 does checks not to overwrite later builds of the tsection.dat and the shapes included (I've outlined how it works in the documentation).

4) Installation
The RE list is only a problem for the minority building routes and there are tools to get around this. End users does not care about this problem as they are not aware of it. I don't see why you need to install the tsection.dat separately, again the XTracks installer does this for you if you don't have a later build already.

I too think the process could be simplified but I do not agree on the suggested solution. The suggested shape.dat is exactly what I send to David when I have a set of new sections I want added but this file has to be scrutinized to avoid duplicate names and to verify the used ids (there has been examples of developers using unauthorized ids).

I don't think the suggested utility is a good idea (or it is but used in a different way). Many end-users would not know what to make of it and having another tool for them to run before they can use the route will make those routes less attractive. Don't push the problem towards the end-user and have him doing more things. There are those that will screw it up no matter how simple it is, those that will not understand what they are supposed to have and those that simply won't download yet another tool.
As Matt says there are other utilities that change around tsection.dat files but there's the same problem with those. Some people use them but many don't and if you ask the average end-user many will not even know what they are.

Those that should take a larger part of the work are the track developers and (possibly) route builders. Leave the end-users out of the problem.
Also I think the administrator should stay on and be the one source for new releases for reasons of quality and order but the work could be much simplified.

Now here is how I propose to have it working:
One central administration that issues ranges of ids for the developers and does the final quality check before releasing new builds of tsection.dat. This quality check could be largely automated to make it easy and the same program could be used by the developers before submitting their new shapes.

No issuing of test builds to developers, you make your own test tsection.dat while developing new track sections.

When a developer is finished with a set of track sections for release he takes the current build of tsection.dat, adds his new entries and changes the build number to "candidate#" or something like that and sends it to the administrator. Administrator receives the candidate and checks it with the tool and by comparing to previous release to see that no other changes have been done (tool could summarize all changes). If all is well the new build is released if not the developer gets a reply with a short description why and then it's the developers task to correct the problem. Since the quality check is much automated it should be quick. If more than one developer sends a candidate at the same time the administrator combines those that are okay and returns the others (again the tool would help with combining the candidates). Come to think about it the candidate file could be a partial file like the shape.dat Tim suggested if there is a tool merging them into a tsection.dat.

I want to keep the central administration for the sake of quality and simplicity for the end-user. With a tool as outlined above and better quality control by the developers (possibly using the same tool) the administrative task should be quite small since we skip the test builds. Better track add-on installers and tools aimed at route builders can take care of much of the other problems. The release of builds has proved that it works for the last couple years we only need to simplify the administration.

Working as suggested above I could imagine taking on the task of administrator had it not been I'm deep into XTracks. People have a hard time holding them apart as it is. I think ideal would be if a well known and respected MSTS site could take on the task (hint Matt) possibly with some help behind the lines.

Comments?

[timbooth]

Thanks for your comments Okrasa, they are much appreciated and I understand your concerns about my idea.

I know that a replacement for David has be found, so my concerns have eased somewhat - but I still feel there is room for improvement.

I admit that an end-user utility is not ideal, though I still feel a utility would be of benefit to advanced users to make use of shapes not yet included in the current official tsection.dat build.

I'm quite happy to retain the tsection.dat build issuing, if the process can be speeded up (and the workload reduced, for everyone involved).

A central administrator is essential, looking after issuing of ID's, ID usage, build checking and issuing. I think developers should be responsible for ensuring their entries are tested before requesting them to be included in the next build. I know David/Steve checks for syntax errors and section/shape ID clashes, but he also check that rules of the tsection.dat have been applied (ie. left and right hand curve sections together, and in right order). Really, it should not be necessary for the administrator to check each entry as that means the administrator needs the full technical knowledge of tsection.dat - which they may well have, but does mean someone less experienced could miss errors that cause problems and delay releases. He also checks to see if the sections are being used efficiently, ie. there's no need to create a new TrackSection entry if an identical one already exists, or could be made via two or more existing ones.

I have done some testing today, and I parsed and loaded the latest tsection.dat into a database on my web server. This now means I have the data in format far easier to check, though a bit more difficult to convert back (I have done that successfully, but not in unicode format yet). It also means I can parse and check new entries in the same way, by pasting into an online form. However, it would easily be possible to input new shapes using some kind of online 'wizard' form - instead of typing in new entries, TrackShapes and TrackSections are configured using a simple interface. Alot of options in the shape entry can be done via tick boxes (TunnelShape, RoadShape etc.), whilst some parameters can be worked out from the form entries - such as NumPaths, and XoverPts). This system could even enter left and right curve entries together, left followed by right, so that rule cannot be broken by developers - the same applies for skews, all though rarely used. The system could also suggest existing TrackSection entries that can be used for a new shape (eg. by entering the require length, or radius/angle), either individual TrackSections or by adding two or more TrackSections (eg. 27m can be made with 25m+2m, or 20m+5m+2m, etc) - this would encourage developers to avoid creating more and more TrackSection entries.

Whether pasted in, or created via a wizard, once new entries are in the database they can be verified and compiled into a new tsection.dat easily.
The administrator can easily see whats been uploaded, but also easily check on allocation usage and TrackSection re-use. Once the administrator is happy with the new entries, they can approve the new entries and a tsection.dat is issued - and also made available from the same server (making it easier for people to download a copy).

In theory, developers (and maybe end-users) could also download an interim development tsection.dat, with the new shapes included (but not approved) - as the shapes are already validated, there is little risk of any problems when using the new entries.

My next task is to create the online shape entry form so that I get developers to test it out with new entries - either by pasting-in or using a wizard. A copy feature would also be useful, so that a developer can create a copy of an existing shape and alter it - however the system would prevent you from trying to add duplicates of existing TrackSection or TrackShape entries within your own allocations.

I will also set up some admin features so that allocations can be issued/revoked and also analysed for usage.

The other advantage of it being online is that registered developers can be alerted when a new tsection.dat is released, or when their allocations have been changed.

Just to prove I have actually got the data in a database...

This table shows how many TrackSections have been allocated and used. I've not verified every single user, but it appears to be correct for me and a few others at least. -updated-
+--------------------+-----------+------+------+
| owner | allocated | used | free |
+--------------------+-----------+------+------+
| Bruce Bridges | 10 | 4 | 6 |
| Bruce Bryant | 300 | 0 | 300 |
| Christopher Cyko | 100 | 10 | 90 |
| Clemens Hanel | 70 | 0 | 70 |
| default | 333 | 195 | 138 |
| Dietmar Jlich | 50 | 28 | 22 |
| Ernesto | 50 | 0 | 50 |
| Etienne Bruere | 50 | 19 | 31 |
| Europeanbahn | 1000 | 0 | 1000 |
| Graham Pitt | 12 | 7 | 5 |
| Juergen Romanowski | 50 | 0 | 50 |
| Marc Nelson | 500 | 245 | 255 |
| Martyn T. Griffin | 30 | 10 | 20 |
| Michael Doms | 200 | 0 | 200 |
| Okrasa Ghia | 326 | 182 | 144 |
| Paul Andrews | 100 | 0 | 100 |
| Peter Gaydos | 50 | 0 | 50 |
| Ralf Riemer | 100 | 37 | 63 |
| Ray Collins | 200 | 0 | 200 |
| Ren Wiggerts | 70 | 0 | 70 |
| Renzo Grassi | 200 | 0 | 200 |
| Rick Grout | 100 | 0 | 100 |
| Roland Bal | 50 | 25 | 25 |
| Runar Oudmayer | 250 | 0 | 250 |
| Steven Masters | 420 | 155 | 265 |
| Teemu Saukkonen | 50 | 1 | 49 |
| Tim Booth | 1788 | 231 | 1557 |
| Tim Bridge | 181 | 72 | 109 |
| Tonny Neijzen | 300 | 0 | 300 |
| TrainSim Creations | 1000 | 0 | 1000 |
| William Adams | 150 | 65 | 85 |
+--------------------+-----------+------+------+

TrackShape ID's:
+--------------------+-----------+------+------+
| owner | allocated | used | free |
+--------------------+-----------+------+------+
| Bruce Bridges | 10 | 4 | 6 |
| Bruce Bryant | 300 | 0 | 300 |
| Christopher Cyko | 100 | 84 | 16 |
| Clemens Hanel | 300 | 0 | 300 |
| Dietmar Jlich | 50 | 29 | 21 |
| Ernesto | 50 | 0 | 50 |
| Etienne Bruere | 100 | 100 | 0 |
| Europeanbahn | 1000 | 290 | 710 |
| Graham Pitt | 32 | 32 | 0 |
| Jeff Rice | 20 | 11 | 9 |
| Juergen Romanowski | 100 | 0 | 100 |
| Marc Filella | 50 | 0 | 50 |
| Marc Nelson | 1700 | 722 | 978 |
| Martyn T. Griffin | 30 | 25 | 5 |
| Michael Doms | 200 | 0 | 200 |
| Okrasa Ghia | 388 | 316 | 72 |
| Paul Andrews | 150 | 0 | 150 |
| Peter Gaydos | 50 | 0 | 50 |
| Ralf Riemer | 100 | 68 | 32 |
| Ray Collins | 200 | 0 | 200 |
| Ren Wiggerts | 70 | 36 | 34 |
| Renzo Grassi | 200 | 0 | 200 |
| Rick Grout | 100 | 0 | 100 |
| Roland Bal | 150 | 147 | 3 |
| Runar Oudmayer | 300 | 0 | 300 |
| Steven Masters | 1400 | 1093 | 307 |
| Teemu Saukkonen | 50 | 11 | 39 |
| Tim Booth | 1768 | 710 | 1058 |
| Tim Bridge | 182 | 94 | 88 |
| Tonny Neijzen | 300 | 0 | 300 |
| TrainSim Creations | 1000 | 0 | 1000 |
| William Adams | 150 | 86 | 64 |
+--------------------+-----------+------+------+

These are based on the allocations listed in the tsection.dat, which I assume are correct.


I've also got some scripts that highlight duplicate TrackSection entries (ie. where more than one straight sections have the same length, and gauge, amongst the developers own allocations) - that includes myself

sections = number of sections that share the same length and gauge
+-----------------+-------+---------+----------+
| owner | gauge | length | sections |
+-----------------+-------+---------+----------+
| Teemu Saukkonen | 1.5 | 100 | 4 |
| Tim Booth | 1.435 | 4.5 | 3 |
| default | 1.5 | 100 | 2 |
| Tim Bridge | 0.75 | 5 | 2 |
| Marc Nelson | 1.5 | 30 | 2 |
| Tim Bridge | 0.75 | 50 | 2 |
| Okrasa Ghia | 0.9 | 24 | 2 |
| Teemu Saukkonen | 1.5 | 50 | 2 |
| Tim Bridge | 0.75 | 10 | 2 |
| Tim Bridge | 0.75 | 125 | 2 |
| default | 1.5 | 50 | 2 |
| default | 1.5 | 1.51059 | 2 |
| Tim Bridge | 0.75 | 25 | 2 |
| default | 1.5 | 20 | 2 |
| Tim Bridge | 0.75 | 250 | 2 |
| Okrasa Ghia | 0.9 | 12 | 2 |
+-----------------+-------+---------+----------+

There are no duplicated curves.

Whilst the above are minor, we need to avoid using too many TrackSection ID's. I think this shows the good work David has done in auditing all the sections.

At the moment I'm just doing this work to see if its possible and allow the people involved to test it out, and whilst I'm happy to retain the system on my web server (for free), I have no objection to Matt, or anyone else, looking after it - if other people feel thats more appropriate

[NeutronIC]

If you want to look after it Tim you go right ahead - Let me know a URL once it's publicly consumable and i'll link to it and send people over I certainly don't need extra work at the moment if someone else is prepared to do it

Sounds great though, good to know this imho crucial part of the sim is staying well managed

Matt.

[OkrasaGhia]

Tim, while I can appreciate your effort with that database of yours I still think a tool working directly on the tsection.dat would be nice. Naturally one does not prevent the other. If I can find time I might put something together.
Most of the checks you mention and some other I have in mind could be made by the developer using a tool, no need to involve the administrator in the early stages of development other than for id ranges.

Good to hear David got a replacement, guess I will learn who this new person is when I submit my new shapes. The administrative task would be much easier if this whole business with test builds was scraped. Developers can make their own test builds as I have been doing all along. You only submit when you're finished and then it is to get a new release.

[Horgy]

I'm not big on this T Section.dat lark, but some programs I notice issue "auto updaters" - Windows is a fine example. Combined with Tim's idea, if this program was issued to end users, it could search the online database and download the relevant entries. Bandwidth depending, it could also download track shapes as individual pieces - reducing the download size of most packages.

Horgy

[timbooth]

Tim, while I can appreciate your effort with that database of yours I still think a tool working directly on the tsection.dat would be nice. Naturally one does not prevent the other. If I can find time I might put something together.
Most of the checks you mention and some other I have in mind could be made by the developer using a tool, no need to involve the administrator in the early stages of development other than for id ranges.

Good to hear David got a replacement, guess I will learn who this new person is when I submit my new shapes. The administrative task would be much easier if this whole business with test builds was scraped. Developers can make their own test builds as I have been doing all along. You only submit when you're finished and then it is to get a new release.

The idea of my proposed online system is so that developers enter/submit new entries there, which are then automatically verified against all the rules and checks, and then the administrator can integrate them into the next tsection.dat build once tested. Its a complete online management system for the standardized tsection.dat, not really developer tool to help developers produce tsection.dat entries.

A tool to check for syntax errors is fine, but does not help the administrator to verify that ID's are valid and being used correctly and efficiently. They would still have to process and check all entries, and would not necessarily know if syntax errors have already been checked for.

I've almost completed the entry verification scripts, so that developers can paste in new entries and have them checked against all the rules (at least the ones I've done so far).

If other developers, and the administrator, don't feel the need for a central online system to manage tsection.dat then I won't take the idea any further. I was just hoping to make the process more efficient for all involved.