Requesting help in improving TIA emulation in Stella

[+stephena]

well there goes my excuse for a new drive

 

Hmm, sorry about that

[Omegamatrix]

I tried to build a 64bit version since I'm using Win7 64bit. It failed on the sdl, and I got some warnings too. I went back and "C:\Users\Jeff\Source\sdl\lib64;$(LibraryPath)" to "C:\Users\Jeff\Source\sdl\lib32;$(LibraryPath)", and it says it built successfully, although I can't verify it since it is a 32 bit version.

 

 

So what am I doing wrong with the 64 bit? Also FYI I was able to set the VC++ Directories no problem just like you laid out. After that I rebooted my computer, and found I could no longer get to them through ALT-F7. There's also wrenches all over the place that you could click which would bring up a tab with the VC++ Directories listed, but clicking on it I got some message that they were no longer depreciated or something. Then it told me the hit the helpful question mark, and that lead me to some Microsoft page telling me I just had to go View, Property Manager. Yet, I found that I have no Property Manager under View.

 

Finally I was able to get into these VC++ Directories by going Debug, Stella Properties.

 

 

Anyhow, here is my outputs, and my Stella build. I would like to be able to build the 64 bit though. I cleaned and rebuilt the 32 bit version, and it seemed to give me two common warnings:

 

>c:\users\jeff\documents\stella\src\emucore\thumbulator.cxx(512): warning C4146: unary minus operator applied to unsigned type, result still unsigned

 

>FBSurfaceGL.obj : warning LNK4075: ignoring '/EDITANDCONTINUE' due to '/SAFESEH' specification

 

 

 

 

I would love to be able to build Stella, I don't really see myself working on it right now (super busy), but at some point maybe. I did try to get a build environment a few years back, but was unsuccessful and moved on. Sort of like TJ, I just wanted to get it working right away. I have used Eclipse in the past and enjoyed it, but I'll take any build environment I can get. This VS 2012 didn't seem to bad, besides the Directories disappearing on me.

builds.zip

[Thomas Jentzsch]

Success!

 

Got Debug and Release for 32 and 64-bit compiling and running!

[Thomas Jentzsch]

I tried to build a 64bit version since I'm using Win7 64bit. It failed on the sdl, and I got some warnings too. I went back and "C:\Users\Jeff\Source\sdl\lib64;$(LibraryPath)" to "C:\Users\Jeff\Source\sdl\lib32;$(LibraryPath)", and it says it built successfully, although I can't verify it since it is a 32 bit version.

 

 

So what am I doing wrong with the 64 bit?

 

I think I had similar initial problems:

• For building and running the 32-bit version you need the path to point at lib32 and you have to copy the 32-bit SDL.dll into the Release and Debug folders.
  
• For building and running the 64-bit version you need the path to point at lib64 and you have to copy the 64-bit SDL.dll into the x64\Release and x64\Debug folders.
  

[Thomas Jentzsch]

I wrote a small test program to address the Pole Position problem (changing NUSIZ0 while displaying the missile). And it turned out the delay is indeed variable depending on the previous size of the missile.

 

From the tested emulators (Stella, z26 and MESS), MESS comes closest by far, but it is still often off by one pixel and missing some subtle differences.

 

Each file contains 3*5*6 = 90 tests.

• The 3 groups are switching m0 size to the size represented by the last two digits of the filename coming from the 3 remaining sizes.
  
• Within each group the cycle for switching the size is tested at 5 positions
  
• Finally the missile is tested at 7 positions
  

Since MESS is pretty close, I have taken screen shots there and marked missing pixel with red and superfluous pixel with green.

 

Now I have to make some rule out of that.

test_x_00.bin

test_x_01.bin

test_x_10.bin

test_x_11.bin

Edited February 23, 2013 by Thomas Jentzsch

[Thomas Jentzsch]

OK, that was (too!) easy. By changing the delay to 1, the missiles works perfect (and Pole Position too). But now that screws the players!

 

Stephen, do you know how I can adjust the delay for missiles and players separately?

[Thomas Jentzsch]

OK, I think that is what mySuppressPx is meant for. Those are clearly set wrong as far as I can tell.

 

But changing something here seems a bit risky without testing a lot. Do you have some more test ROMs, Stephen? Some who display correctly but do weird things with NUSIZx and/or RESyx?

[Thomas Jentzsch]

I think NUSIZ0 (and NUSIZ1) should be changed into this:

   case NUSIZ0:  // Number-size of player-missle 0
   {
     updateFrame(clock + 1);
     myNUSIZ0 = value;
     break;
   }

Changing the delay to 1 fixes the missile problem in Pole Position and removing mySuppressP0 = 0 (which AFAIK should only be used in case of RESxy) fixes Yathzee.

 

Meltdown becomes worse (right two copies are completely gone now), but I think that's from a more general problem with changing NUSIZx while displaying a player copy.

 

The problem with the players appearing at the left are clearly resulting from resetting mySuppressPx at the end of a scan line. But I yet have no idea, how to determine if the suppressed copy had already been drawn.

 

For that I need to understand the tables much better. Some comments why which value is set under certain conditions would sure have helped.

 

Maybe one can calculate the duration of mySuppressPx when setting it?

 

[+stephena]

I tried to build a 64bit version since I'm using Win7 64bit. It failed on the sdl, and I got some warnings too. I went back and "C:\Users\Jeff\Source\sdl\lib64;$(LibraryPath)" to "C:\Users\Jeff\Source\sdl\lib32;$(LibraryPath)", and it says it built successfully, although I can't verify it since it is a 32 bit version.

 

 

So what am I doing wrong with the 64 bit? Also FYI I was able to set the VC++ Directories no problem just like you laid out. After that I rebooted my computer, and found I could no longer get to them through ALT-F7. There's also wrenches all over the place that you could click which would bring up a tab with the VC++ Directories listed, but clicking on it I got some message that they were no longer depreciated or something. Then it told me the hit the helpful question mark, and that lead me to some Microsoft page telling me I just had to go View, Property Manager. Yet, I found that I have no Property Manager under View.

 

I think following these directions from Thomas should work; they are exactly what needs to be done:

 

I think I had similar initial problems:

• For building and running the 32-bit version you need the path to point at lib32 and you have to copy the 32-bit SDL.dll into the Release and Debug folders.
  
• For building and running the 64-bit version you need the path to point at lib64 and you have to copy the 64-bit SDL.dll into the x64\Release and x64\Debug folders.
  

 

Finally I was able to get into these VC++ Directories by going Debug, Stella Properties.

Anyhow, here is my outputs, and my Stella build. I would like to be able to build the 64 bit though. I cleaned and rebuilt the 32 bit version, and it seemed to give me two common warnings:

 

>c:\users\jeff\documents\stella\src\emucore\thumbulator.cxx(512): warning C4146: unary minus operator applied to unsigned type, result still unsigned

 

>FBSurfaceGL.obj : warning LNK4075: ignoring '/EDITANDCONTINUE' due to '/SAFESEH' specification

 

Don't worry about those warnings for now; I get them too. The first one will disappear when we move to the new ARM emulation (post 3.9, if there is one). And the second is something I haven't gotten around to fixing yet.

 

You do realize that you're able to run the 32-bit version directly in 64-bit mode, right? Now while it would be nice to get it all working (and the above should do that), don't be discouraged if you can't get it working right away. From the POV of development, using the 32-bit version would be fine.

[+stephena]

All of this confusion in setting up VS2012 indicates to me that I need a webpage/something in the manual that clearly explains how to do it. I've never really bothered before, since nobody else was working on it, and nobody asked how to do it. I'm going to concentrate on creating such a page; hopefully I can get something done by the end of the weekend.

[+stephena]

OK, I think that is what mySuppressPx is meant for. Those are clearly set wrong as far as I can tell.

 

But changing something here seems a bit risky without testing a lot. Do you have some more test ROMs, Stephen? Some who display correctly but do weird things with NUSIZx and/or RESyx?

 

First of all, good progress on changing the values. I'd done that in the past, and saw similar 'fixes', which fixed a problem in one place and broke something elsewhere. My main problem was that I don't understand why the delay was 8, and why setting it to 1 fixed it. Where did the 8 come from in the beginning? I assume it was chosen to correspond to how most ROMs work. Or does the 8 have some significance when wrting NUSIZx (like it takes 8 cycles or something for it to take effect??). That's my big problem; I don't know why the values are changing. And because of that, I wasn't sure that (for example) changing to 1 and fixing a ROM but breaking another was the appropriate fix. IOW, which was the more correct solution??

 

With the changes to NUSIZx delay, the following ROMs break for me:

 

battlezone.a26: Extra data to the right of the radar screen

Chun-Li: Corrupted third/right-most 'head' in number of lives

 

I'll try to find more.

 

EDIT: The corresponding code in MESS seems to have a variable delay. That's why 1 works for some ROMs, and 8 for others. I'm willing to bet that there are other ROMs where neither of these values are appropriate. So I think we need to figure out what MESS is doing, and do the same thing. It looks to be calculating the delay based on how much has already been drawn. Problem is, as you say, there's currently no way to determine this. It probably needs to be calculated somewhere in updateFrame(), while the pointer is stepping through each 'pixel'. Or maybe one of the tables can be extended to include the variable delays.

[Omegamatrix]

I got it all compiling now. The big gotcha I ran into was that the VC++ Directories have to be updated for every configuration (Debug - Win32, Release - Win32, Debug - x64, Release - X64). I was also copying the SDL.dll from my 64 bit Stella 3.8 into the 32 bit version, doh!. When I finally peeked into the library folders I realized that there are indeed 32 bit and 64 bit versions SDL.dll right there.

 

 

Anyhow, it seems to work great now!

[Thomas Jentzsch]

My main problem was that I don't understand why the delay was 8, and why setting it to 1 fixed it.

Same here (for now!).

 

Where did the 8 come from in the beginning?

Who wrote that code?

 

I assume it was chosen to correspond to how most ROMs work. Or does the 8 have some significance when wrting NUSIZx (like it takes 8 cycles or something for it to take effect??).

After looking at Battlezone I think I have to write another test file. That should make clear(er) what is going on.

 

Also I think we might run into a general problem if the delays for missile and players are both different and cannot be compensated with mySuppressPx.

 

EDIT: The corresponding code in MESS seems to have a variable delay. That's why 1 works for some ROMs, and 8 for others. I'm willing to bet that there are other ROMs where neither of these values are appropriate. So I think we need to figure out what MESS is doing, and do the same thing. It looks to be calculating the delay based on how much has already been drawn. Problem is, as you say, there's currently no way to determine this. It probably needs to be calculated somewhere in updateFrame(), while the pointer is stepping through each 'pixel'. Or maybe one of the tables can be extended to include the variable delays.

I am a bit worried that the whole current Stella emulation approach might not be tailored to support such special cases. Therefore the code might become pretty ugly or that we need really huge tables.

Edited February 23, 2013 by Thomas Jentzsch

[+stephena]

Brad Mott wrote the initial TIA implementation, and used the approach of tables for (I think) speed on slower systems. Stella was originally designed to run on a 486.

 

I am a bit worried that the whole current Stella emulation approach might not be tailored to support such special cases. Therefore the code might become pretty ugly or that we need really huge tables.

 

The code doesn't have to be added to tables. In fact, some of the changes I've made over the years are to move away from tables. You'll see this in the HMOVE stuff in TIATables; the code is commented out completely, since the timing was better suited to be calculated at runtime.

 

I'm still thinking that there's nothing wrong with the mask approach. After all, it just basically specified a bitmask for the serial output for the players, etc. I think what needs to be tracked is the clock at which the player starts drawing, the clock at which the first copy has passed, and. for each clock thereafter, compare the two. Once they've passed, suppression will be turned off. I think this is what the original comment about reseting suppression "being too slow" was about. I'm convinced that if this was done, the current mask approach would work.

 

So, what we need to determine is (a) when/where in the code the clock should be set for start drawing, and (b) where the comparison for 'passing the first copy' should be happening. Notice that this is basically what MESS is doing with the following structure:

// Per player graphic

// - pixel number to start drawing from (0-7, from GRPx) / number of pixels drawn from GRPx

// - display position to start drawing

// - size to use

struct player_gfx {

int start_pixel[PLAYER_GFX_SLOTS];

int start_drawing[PLAYER_GFX_SLOTS];

int size[PLAYER_GFX_SLOTS];

int skipclip[PLAYER_GFX_SLOTS];

};

 

The size and 'skipclip' are, I think, taken care of in the masking. The start_pixel and start_drawing correspond roughly to what I talk about above.

 

EDIT: I should be clear here, that I'm asking whether this makes sense, and not saying for sure that it will work. This is where my knowledge of the TIA breaks down, and where I need clarification. Does what I'm saying even make sense??

[Thomas Jentzsch]

As far as I can tell this makes sense for normal cases, but when I look at my experiments with odd coding, there just so many variables to consider, that I suppose the table approach will soon become impossible. IMO. Without having a real deep understanding of the existing code, I think instead of defining the state of each pixel from scratch, it might be a better idea to emulate the objects which are on screen (like MESS does). That should make it easier to react to uncommon code.

 

The question is, what do you really want to achieve?

 

Do you want to fix the existing problems of real games (and maybe demos)? That can be maybe done by tweaking the existing code. But then there is no guarantee that the next demo or game will be working 100% correctly.

 

Or do you want to to create an emulation which works 100% correct? Then I suppose a rewrite from scratch will be most attractive for a programmer. The existing code is not very well commented and (like you say) more optimized for speed than for 100% accuracy. Trying to convert this might result into even more, often very tedious and unsatisfying work (as you know already ). If you go for this choice, I would opt for finding someone who is interested into core emulation without having to care for all user interface etc. stuff. Those people exist.

E..g http://code.google.c...ri/2600/?r=4097 or http://sourceforge.n...s/emu7800/v1.5/

 

Again, that's only my first impression without looking deeper into the code, maybe you have more insight and disagree.

Edited February 23, 2013 by Thomas Jentzsch

[+stephena]

As far as I can tell this makes sense for normal cases, but when I look at my experiments with odd coding, there just so many variables to consider, that I suppose the table approach will soon become impossible. IMO. Without having a real deep understanding of the existing code, I think instead of defining the state of each pixel from scratch, it might be a better idea to emulate the objects which are on screen (like MESS does). That should make it easier to react to uncommon code.

 

I'm forced to agree. The complexity of the current code is just getting too overwhelming.

 

The question is, what do you really want to achieve?

 

Do you want to fix the existing problems of real games (and maybe demos)? That can be maybe done by tweaking the existing code. But then there is no guarantee that the next demo or game will be working 100% correctly.

 

I want the emulation to be 100% correct, so I never have to look at this part of the code again I'm not interested in cheats that work for certain cases; that was what was fixed with the HMOVE changes in version 3.0. If there are fixes/cheats like this, we'll be constantly tinkering with it, which is precisely what I want to avoid.

 

Or do you want to to create an emulation which works 100% correct? Then I suppose a rewrite from scratch will be most attractive for a programmer. The existing code is not very well commented and (like you say) more optimized for speed than for 100% accuracy.

 

Yes, but the question is, will a rewrite be easier than fixing what's there, and will I have any help doing it? This isn't the first time that the complexity in the TIA code has been brought up. Others have taken a stab at it in the past, and basically walked away. And I'm at that point myself. If a rewrite would make others more willing to stay on and help me, then I'd strongly consider it.

 

Trying to convert this might result into even more, often very tedious and unsatisfying work (as you know already ).

 

This. A 100 times this.

 

If you go for this choice, I would opt for finding someone who is interested into core emulation without having to care for all user interface etc. stuff. Those people exist.

E..g http://code.google.c...ri/2600/?r=4097 or http://sourceforge.n...s/emu7800/v1.5/

 

The problems are (a) does that code do any better than Stella currently does, and (b) will anyone want to help. Or in this specific case, would you be willing to stay on?? The first project you mention has extremely easy-to-follow code. But there are several places where it says 'get rid of magic numbers'. IMO, these magic numbers may be the corner cases, exactly the types of issues we're trying to fix right now.

 

Again, that's only my first impression without looking deeper into the code, maybe you have more insight and disagree.

 

My only concerns, really, are (a) not burning out, (b) having some people come on to help me in a sustained effort, © not having them burn out, and (d) not doing a whole bunch of work and being worse off than we are right now.

 

I kind of agree that the table approach may be reaching its limits. On a different note, implementing RSYNC support has come up in the past, and I have absolutely no idea how to integrate it with the current approach. But I do have some idea how to do it with the new approach.

 

I just want to get the TIA working 100%. I don't really look on it as something that should require the bulk of my time. From my POV, it's just another component, like the 6507 or 6532. It should be completed, so I can move on to debugger improvements, user-interface stuff, movie recording, TV effects, or in other words, general end-user improvements. That's the area I feel Stella excels over other emulators. The TIA support has been weaker for some time; I just want it done and over with. And that's kind of why I'm proposing moving on from the project if it can't be fixed. Other emulators are doing better wrt TIA emulation. If it can't be fixed, maybe it's better to focus development on other emulators. I don't want to, but every project reaches its limit at some point.

 

What are your feelings on this??

[Omegamatrix]

Not to deflect this topic from updating NUSIZx, but I implemented a little code for RSYNC. It fixes RSYNC so that Extra Terrestrials is now correct, and my test rom now works. I wrote some comments in the code explaining what is still not being correctly handled. Although I tried to keep it short the comments were longer then the code itself.

 

 

I don't know if I've broken anything else, and I didn't want to try and submit anything to the repository until I know more about the correct procedures. I've used Perforce before, and the company I was with had a branching strategy for development code, prototype code, and release code all coming off the main line. Once the code was thoroughly tested it would get merged back to the main branch, but not until then.

 

 

I included the updated TIA.cxx file, and some pictures on my TV with emulator snapshots.

 

 

StellaRsyncUpdate.zip

[Thomas Jentzsch]

My feelings are, that it would be a complete shame to abandon Stella, even with some TIA flaws. For me it is not only just another Atari 2600 emulator, but a very valuable development tool. And that makes the big difference to all other Atari 2600 emulators. Also your almost immediate support for e.g. new hardware has always been much better than I would expect from anyone else.

 

Whenever anyone is trying new TIA tricks, he should never trust any emulator, but test on real hardware. This would still be true after a "100% perfect" TIA rewrite, since all new tricks are untested by definition. And the existing TIA documentation can have errors too. So you will never be 100% safe from TIA fixes, only the chances of the emulator to work correctly with those new tricks become better.

 

Regarding the rewrite, I could try to help with testing ROMs, writing test ROMs, check on real hardware etc. I might even be able to help with the coding once the basic structure has been set up, but I don't think I can (and want to) do the lead in the TIA rewrite. I have zero experience with writing emulators and my interests and strengths are somewhere else.

 

But since this idea had been brought up in the Future of Stella thread (where I killed it ), maybe you should ask there again.

[+stephena]

Thomas, I just did a preliminary check of the BizHawk emulator mentioned above (I've already played with EMU7800 in the past), and while it works for basic ROMs, it tends to fail on many others. And I recognize the failures as things present in other emulators, and stuff that was fixed in Stella some time ago. So I think that moving to those codebases would be a step backwards. The thing that makes me think Stella can still be salvaged is that (a) it already fixed most of the omissions in other emulators, and (b) playing with different delay values fixes the problems we currently have. This makes me think that the code itself isn't deficient, it just isn't tracking timing as precisely as it should. A similar issue occurred with the HMOVE functionality. Once I moved from static delays (in TIATables::CompleteMotion) to runtime calculations, the rest of the code just automatically started working. So it wasn't the player/missile tables causing the issue, just my usage of them. I think something similar will happen when player timing/suppression is fixed.

 

As for never reaching 100% perfection, I probably should reword that by saying that I at least want to match what other emulators are doing. If something doesn't work anywhere (like the RSYNC stuff did up until a moment ago), then I have no issues with it. It needs research to be completed. But when Stella doesn't properly emulate things that almost every other emulator does, then it becomes a problem for me. I guess there's definitely a bit of a pride thing going on there

 

While I think a rewrite could make some things easier, I personally don't think I'll get the manpower to do it. You state that you don't have the time or desire (and believe me, I absolutely understand), and frankly neither do I.

 

I'd appreciate if you have the time, to continue what you're doing. Your test ROMs determined that a delay of 1 vs. 8 fixed certain things, as did not turning off suppression on NUSIZx write. You later inferred that this should be done in RESPx instead. This is invaluable information, and it would be great to get more insights and test ROMs related to this. You don't necessarily need to understand every line of code to be helpful; your TIA knowledge is more helpful in this case. For example, the original delay of 8 for NUSIZx, which is also the baseline value in MESS (it's 8 at most, and decreases from there). What is the significance of this 8? Knowing that, and being able to document it in the code would be extremely valuable.

[+stephena]

Not to deflect this topic from updating NUSIZx, but I implemented a little code for RSYNC. It fixes RSYNC so that Extra Terrestrials is now correct, and my test rom now works. I wrote some comments in the code explaining what is still not being correctly handled. Although I tried to keep it short the comments were longer then the code itself.

 

Well, that kills my opinion that I have no idea how to integrate RSYNC functionality with the current code I look into getting this added right away. Even if it isn't 100% (the 'real time effects' you mention), it's still great to have.

 

EDIT: Also, don't apologize for comments being longer than the code. If there were more comments there, we probably wouldn't be having the issues we're having now

[Thomas Jentzsch]

For example, the original delay of 8 for NUSIZx, which is also the baseline value in MESS (it's 8 at most, and decreases from there). What is the significance of this 8? Knowing that, and being able to document it in the code would be extremely valuable.

I have no idea too. I could not make any reference to Andrew Towers' notes.

 

It is either resulting from trial and error or the simple result of some more complicated math described in Andrew's notes. I guess it is the former, just because Andrew mentions only NUSIZx very briefly. But I am not hardware guy, so I have a hard time following the notes.

[Thomas Jentzsch]

Looks like Omegamatrix is already ahead of me understand the TIA emulation.

[+stephena]

Sorry for the flurry of messages, but I realize it seems like I'm flip-flopping on my position. So let me be clear:

1. I want the path of least resistance. That means figuring out whether it's easier to start from scratch or fix what is present. I always tend towards the latter in my professional work, since older code, unless it is complete crap, is usually well tested and somewhat ugly for a reason; it takes care of many corner cases. And rewriting means not only fixing what the current code doesn't do, but making sure it still does everything the current code does do.
   
2. I need to balance (1) with the desire to bring others on board. Obviously if the stuff is so complex that I can't get anyone to help, that makes more work for me and tilts in favour of a rewrite. But when every other project I look at either has deficiencies that Stella has corrected, or isn't compatible with the Stella license, then the amount of work required increases exponentially.
   

This is what's stressed me out over the past year. I can't see a rewrite being feasible, and I can't get anyone to help me tackle the current code. The best I can hope for is studying the current code and figuring out why certain things are done a certain way, and perhaps come up with test ROMs to back this up. Extra commenting alone would probably make it possible for me to do the coding myself.

[Thomas Jentzsch]

I would be flip-flopping here too.

 

Maybe the current emulation is THAT close to (almost) perfection. Just we don't see it. Or maybe it is far off. I don't know.

[+stephena]

Not to deflect this topic from updating NUSIZx, but I implemented a little code for RSYNC. It fixes RSYNC so that Extra Terrestrials is now correct, and my test rom now works. I wrote some comments in the code explaining what is still not being correctly handled. Although I tried to keep it short the comments were longer then the code itself.

 

The easiest thing to do in this case is send me a diff. If you've grabbed the code from SVN, then it would be as easy as 'svn diff', or something similar in the UI-based version of SVN. I'm in the process of documenting this in a developers howto webpage.

 

Also, I notice your code is almost exactly the same as WSYNC, except you also add a check for >22. What is the significance of this 22??