Atomic Systems

Hello all, I'm going to try and make this post short, but in reality I've got a lot to say so there's only so much I can cut out.

First of all, we all know there are a multitude of flaws within the TS engine, and I'm certainly not going to pretend otherwise, but what I am attempting to do will hopefully treat a symptom as we can't address the cause.

I was writing some scenarios for the SVR route for SSS but kept getting set back by tempdumps whilst adding static or player stock throughout the scenario. I have placed realistic static items where possible and worked hard to create the 6 running rakes of operational stock currently available where stock permits (thanks very much to Gordon Mack/Gopher here for so much of it). Then I have taken the WTT available from SVRLive and started making parts of the whole day of operation for Class 52 D1015. Even so, I have had to turn down the scenery quality (which is texture resolution related) to the 2nd from bottom setting to get the scenario to load - but still in a state where crashes can sometimes occur in either form of editor.

The main thing that struck me about the route was the daunting quantity of downloads required from here in order to function. I fully understand why it has to happen, but it doesn't make it any easier - all of the zip files I downloaded totalled 2.5Gb when compressed!!! This in turn leads me to believe that the relevant ticks for all of these assets is what is behind the tempdumps I am getting - the rolling stock is merely the straw on the large camel's back.

So, with the aid of RW Tools I got a CSV file listing all of the items used by the route in its entirety - after removing duplicates this came to 1038. The amount of blueprints I have installed for just the UKTS freeware packs comes to over 3,000. Further investigation found situations like Brendan\IrishRail - which contains over 1,000 assets, but the SVR uses around 18 of them. Once I was armed with the list of blueprints used by the route I set about deleting the DTG/Kuju/RSC ones from the list due to DRM or .ap file, and then added new folder paths to include a new custom set of common folders to a duplicate of the original file path, in this case SteamSoundsSupreme\SVR_3rd_party and when the file list was duplicated and modified I made a batch file out of it with a few uniform amendments to the lines. This copied and created all of the Asset folder structures and this all happened fine.

I made a clone of the Route in RW Tools as I wanted to be able to make direct comparisons between performance of the vanilla version compared to my diet caffeine free one. The issue came with needing to repath all of the references to the assets within the relevant files. There was no easy way to do this other than using the RAW Tools list of assets used, and selecting each individual asset and pointing it to the new blueprint - just remember there are still over 1000 unique assets that I needed to do this too! I didn't fancy that too much so instead opted to use the "Find in multiple files" function with bespoke arguments for the two windows depending on which asset I was repathing. Having spent 5 hours doing this this morning, after completion I ran the route through RW Tools "Check route" routine to look for items that were considered missing. It only returned a few which match what is missing from the default SVR beta, and anything RSC/DTG/Kuju which I hadn't messed with. I have a feeling this approach may have led to some errors creeping in, whereas using RW Tools method is less fallible because you click on the actual blueprint file required in its new location.

I also edited the routeproperties.xml file to remove all of the dependency ticks for the items that I had just created the duplicate blueprints for. I also edited the location of the default terrain textures and where the route template was stored also at this time.

When I fired up the route I got a lot of visibly missing items, but it's important to add that it was my own scenario and this loaded with no errors and started as it should. So the AI could get properly lathes etc, but visually I had a lot of items missing that RW Tools is not reporting as missing. Below is a screenshot of what I currently have occuring so you can see for yourself.

I thought I would make a post to see if there's any useful suggestions people may have and also to report if I do get it up and running what difference (if any) is experienced. Although I have this setback, I had turned the scenery quality up by one more notch (so 33% to 66%) and hopefully that is a good sign it's kicking the bad behaviour out - I just need to discover more about what problems I currently have and why. I've also included a screenshot from before I changed the texture path in the routeproperties.xml file.

My main theory at the moment, and I don't believe it has a simple solution (groan) is that within blueprint files, it is possible to have references to other blueprint files. There are 2 ways these appear.

One is with the full file path as one single entry such as below:
For this type of reference it is perfectly fine to reference them from a different blueprint without needing to tick the specified Provider and Product in the tab - this is how a lot of child objects such as headboards are done to save adding ticks to lists of instructions for use.

The second type is more worrying and appears like:
These can be referenced but WILL NOT show if the provider and product listed is not ticked. If, as I suspect, these file references are causing my blank bits in the route then not only do they require moving, they also need to be repathed in the blueprint of the asset referencing them.

The only way I can do this is by looking in every one of the 800 odd blueprints I have copied across and manually check each and every blueprint reference contained within for this type of reference and then make a new list of files to copy AND amend the asset blueprint accordingly.

deltic009 wrote:I was writing some scenarios for the SVR route for SSS but kept getting set back by tempdumps whilst adding static or player stock throughout the scenario.

"Tempdumps" or "Unable to save to . . . " dumps? Tempdumps are generally a specific error (rare) but the latter are an indication of being out of available memory and are far more common when editing.

This is often resolved by disabling the scenery directory of the route whilst editing the scenario as the editor will run using far less memory. Quite often, but not necessarily so if you push to the limit, this will not affect the scenario when played. It certainly varies by route (and I don't claim to know the cause) but I reckon London-Brighton is the worst offender - it is impossible to place any locos/vehicles in some areas (interestingly, not London!).

The other thing you must, must, must do when editing a scenario is save regularly and watch your memory use because TS's memory management is appalling. As soon as TS gets to around 3.5 GB memory in use it will go bang - but quite often you can save and restart and find you are using substantially less memory for having done so.

This reminds me of a program called TrainStore or so, for MSTS. Adding your observations and some recent posts by Matt up takes me to the conclusion that it is time for something like that.

I guess you could reach your target faster if you create a new installation of the game (or Steam) that only holds the Products that are originally required and then you strip these folders from all that is not needed. Matt clearly said that it is not a matter of how many products you have included but of how much unused stuff lies around in all the unused folders.

Of course, the problem of nested references in blueprints persists, but I would assume that you can ship around it by using some guessing. I.e., you leave the folder and all its subfolders intact wherever you see one blueprint referenced. Or you keep all the Simulation folders and just strip out the unused blueprints in Engine folders. These are but naming conventions but many adhere to them, so with some luck, you save a lot of worrying.

I don't think that scenery items will have much dependencies, other than particles for smoke, maybe. Some radical cleaning in those folders will speed up the loading. I believe it is a frequent case where the route builder feels the need for a few items of scenery from a certain pack and now the whole of the scenery collection takes up time and storage capacity in RW's data structures.


Moving blueprints can be a tricky pursuit indeed. There are many nested references. Unfortunately, there are also many cases where links are broken from the start and some of those are meaningless.


Once upon a time, when early RW versions used to crash when some blueprint was not found, I wrote a tool called Blueprint Checker. You still find it here. It is more of a proof of concept. A few people used it, but very few routes were actually salvaged. A contributing factor was that those who destroyed their data structure where those who did not want to dig through endless lists of files. So I shelved the whole project.

You might want to try it though, as I seem to be more obsessed with Tracks.bin than Mike. Most of my tool is just duplicating his work with an inferior user interface, but some of it might proof complementary. And the manual may spark some thoughts on dependencies, maybe.

Thanks very much for your input so far folks, and for the link AndiS I shall be sure to check it out.

Whilst looking at this is a big job, I feel it necessary due to the core software being more or less an unchangeable thing from here-on in. Maybe once the updates have definitely stopped from Dovetail those with some real knowledge can start to see what backing of code can be achieved and advise accordingly.

Fireman44 maybe my post wasn't too clear, I can neither edit nor play the scenario without the scenery quality slider being down at 33% (one from the lowest) and still occasionally encounter crashes - and you're correct it is the unable to save variety of dump out I'm referring to.

For now I shall continue my endeavours, but it sounds like the tool you wrote should be of good help, even if I have to manually repath things each item at a time.

Is anybody else having the same kind of issue(s) with this route?

gptech wrote:Is anybody else having the same kind of issue(s) with this route?

Yes, Steam Sounds Supreme themselves have constant "Unable to save....." and the people behind Golden Goldsmith Scenarios who are trying to produce realistic scenarios for the route.

Also worth remembering I've only just done fresh OS and TS installs on new drives so my installation is very light in comparison to times of old too Gary (135Gb or similar all in for TS).

AndiS wrote:Matt clearly said that it is not a matter of how many products you have included but of how much unused stuff lies around in all the unused folders.

One of the problems is selecting the Kuju folder for an item of stock is all the other items it brings along.

I have personally not found any evidence to suggest what's located in a certain folder structure matters for memory usage, just solely what is actually loaded into the route/scenario.

I am probably wrong, but it is my understanding that each Blueprint.pak file is a cache containing pointers to where the blueprints for that particular Developer/Product (be it assets or routes) are to be found, so that when the program calls for a Geo file, it knows where to go without having to search the whole drive. The Kuju one is usually around 10MB, and a middle size route, such as LivMan, is around 4MB. Not huge in comparison to the RAM available. Having a large number of assets on tiles is more of a memory hogger. The occasional large one is OK, but put 3 or 4 together then the time taken to move from one to another becomes noticeable (stutters).
When building scenarios, I often switch off the scenery. When building routes, I try to avoid choosing a developer/product for just the odd item, unless it is crucial to the route (very few would answer that description).

phat2003uk wrote:I have personally not found any evidence to suggest what's located in a certain folder structure matters for memory usage, just solely what is actually loaded into the route/scenario.

I would largely agree Richard but for so long now it has been known that usage of the Kuju/RailSimulator folder causes issues and yet most route builders probably don't fill a route with items from in there - certainly not more than the actual routes like S&DJR that are based solely around them, and the official routes encounter a lot less problems so it would seem the only differential is that far more providers are ticked in the freeware offerings.

SVR also suffers IMO from the usage of 3D track that not only impacts FPS but also stability, I've never been able to run any of my JT routes with 3D track for example, but with 2D it's fine.

deltic009 wrote:

phat2003uk wrote:I have personally not found any evidence to suggest what's located in a certain folder structure matters for memory usage, just solely what is actually loaded into the route/scenario.

I would largely agree Richard but for so long now it has been known that usage of the Kuju/RailSimulator folder causes issues

This is said but the Wherry Lines for example utilises this folder and is a bastion of reliability as far as I've been able to assert. The only thing that will ever tip it over the edge is using too much high resolution rolling stock.

deltic009 wrote: I would largely agree Richard but for so long now it has been known that usage of the Kuju/RailSimulator folder causes issues and yet most route builders probably don't fill a route with items from in there - certainly not more than the actual routes like S&DJR that are based solely around them, and the official routes encounter a lot less problems so it would seem the only differential is that far more providers are ticked in the freeware offerings.

Many of the "official" routes use Kuju assets, but effectively "clone" them in the same way as you are suggesting in the first post. They are then part of the routes "assets" A typical example is the standard 5m platforms (in some cases there is a fresh Texture file, but the Geo is the same in all other respects). You can have situations in which an object has the same "local name" (ie the one in the menu) but can be selected from 2 or more developer/product combinations (see a typical asset list using RWTools for a freeware route). In some cases, the local name has a code set of letters prefixing it, but not always!

When we started The Friargate Line it used Kuju assets but when we moved to Woodhead as our commercial DLC and removed the Kuju links it took a full 30 seconds less to bring the route up afterwards. As has been said, where more recent routes use Kuju models they've been tweaked so that they arrive as part of that route's .ap file. Yes, it's duplication but it's a price worth paying, IMO.

Keith

Three things to consider:

1) The game may well have internal limits on the sheer number of assets, or become slow or inefficient if the number of items on some list becomes really large. One such list will be the assets found in an included folder, even if most of these items are not loaded to memory. So it is not just the 3.5 GB memory consumption that you can observe, it can also be some internal limitation regarding handling of massive asset lists.

Remember that if you go into World Editor, you do see all these unused assets in the list of stuff to place, so the game must process these blueprints to the extent that it can decide whether it can show it or rather not because it is broken.


2) Everyone's Kuju folder is different, because many freeware creators ignored the official guideline to start a new folder for each project (aka Product). This was not fair game and argued by some that checking the extra box in the list to make the items appear was something that people do not understand. There could have been some other reason in early Rail Simulator / Railworks that I forgot. Anyway, it was customary back then.

Much of that are reskins and these often need to live in the original folder for some links to work - or so was the assumption. Redistribution of original files is prohibited for good reasons, so here we see some solid argument for bloating the Kuju folder.


3) Much of the wannabe Kuju stuff is "first generation". Quite a few people did not really understand quite a few things in the early days. I excuse to any creator feeling offended, but broken links were not uncommon back then. I focussed on RSDL stuff and reported all my findings to them. I am guilty of not doing the same for 3rd party stuff. There was also this fear of discouraging creators by being overly critical in the air, and there is more to do than I can ever get my hands at anyway. It will be the same for most others.

The net result is that everyone may have some assets on their harddisk that have some issues. You practically forget about them.

I found that the game uses a lot of time for handling broken stuff. It is much amplified if you use LogMate, but even without it, you will note it.

Then there seems to be some limit on how many errors the game can handle. When loading a route before I had all requirements installed, I found much missing that was already installed. After I installed some unrelated stuff, those things that had not been altered started to appear. It was about track and terrain, as far as I remember.

The bottom line is that it really pays to have a flawless installation, and maybe a slim one, too. This is what makes this project big. RW became pretty good at working around asset flaws, but it does come at a cost.