35 Responses to “WARNING! Windows Server 2003 SP2 May Destroy Your ISA Firewall without Warning”

1. E-Bitz - SBS MVP the Official Blog of the SBS "Diva" : SP2 Unleashed - back at ya Says:

   March 23rd, 2007 at 7:26 pm

   […] SP2 Unleashed - back at ya http://blogs.technet.com/windowsserver/archive/200...d.aspx Rapid customer adoption of Windows Server 2003 SP2 continues.  In less than a week since release there were more than 400,000 successful downloads!  Interesting that some have commented that we “quietly” released SP2.  The trade press have certainly covered it, since November when we broadly publicized the Release Candidate and product details, and at launch last week. At last count there were several dozen news stories about SP2 over the last week, and many blog posts.  SP2 also required less advance education compared to SP1 or XP SP2, because (by design) it is generally much easier for customers to “consume.” Dear Joel.  I’m sorry … I don’t meant to be mad at you but instead mad at whomever stuck SP2 on Windows update. That “rapid customer adoption” is because you guys stuck it up there on Patch Tuesday without giving us notice. 400,000 downloads and how many SBS boxes that customers now have to pay their var/vap to clean up the gotchas?  http://blogs.technet.com/sbs and see the issues there.  Help and Support.  ISA server.  Broadcom nics again.  SP2 can’t get installed period that I just blogged about. The trade press, the Microsoft update blog, the WSUS blog, no one said that it was coming out on Patch Tuesday after DST.  Where on the Release candidate page last November did you tell me that you were going to release this in March on a Patch Tuesday?  Show me. Easier for customers to consume?  You go patch a SBS box and cross your fingers and see if Help and Support survives the journey will ya?  Try charging a customer for a patch that doesn’t work.  You go charge a customer for googling up why you can’t get it installed, why you helpsvc doesn’t work, why DHCP isn’t working.  Granted this isn’t all the cases, and many do get SP2 installed just fine, but man, I have still way too many pings and dead body reports that I’m rounding up. And for the record that Windows Server blog only had blog posts after it hit the download site.  It was up on Microsoft update on MY server way before it was blogged.  I even asked in the www.activedir.org listserve if it RTM’d and there was confusion there if it RTM’d. I don’t mean to be shooting the messenger Joel, but sorry, I greatly disagree here.  We ITpros didn’t have the heads up we needed for that to come offered up on Patch Tuesday.  That’s the MSRC’s day and should be reserved for their patches and theirs alone.  Service pack 2 could have been released on that day but it didn’t have to be on Microsoft Update on that day. I’m sorry, Joel, you didn’t tell us, your customers like you should have. I apologize for this ranty post but you just hit a nerve. (edit and added these links) Thomas Shinder Blog » Blog Archive » WARNING! Windows Server 2003 SP2 May Destroy Your ISA Firewall without Warning: http://blogs.isaserver.org/shinder/2007/03/23/warn...rning/ You cannot host TCP connections when Receive Side Scaling is enabled in Windows Server 2003 with Service Pack 2: http://support.microsoft.com/default.aspx?scid=kb;...927695 That may be 400,000 downloads, but that’s not 400,000 successful installs. Share this post: email it! | bookmark it! | digg it! | live it! […]

2. Susan Says:

   March 23rd, 2007 at 8:00 pm

   According to the Windows server blog there have been 400,000 downloads of SP2. Of course they don’t say how many “successful” downloads that may have been.
   http://blogs.technet.com/windowsserver/archive/200...d.aspx

   The SBS blog that first blogged on this on the 19th indicate that you may need to get newer NIC drivers as well
   http://blogs.technet.com/sbs/archive/2007/03/19/vp...m.aspx

   (Cross posted this blog link to www.patchmanagement.org and www.activedir.org)

3. W2K.PL » Blog Archive » Windows 2003 SP2 i potencjalne problemy Says:

   March 24th, 2007 at 3:40 pm

   […] Zacznijmy od zgłoszonego przez jednego z użytkowników problemu z własnymi plikami ADM. Wygląda na to, że przygotowane przez nas pliki ADM bez znku końca lini i powrotu na końcu (czyli bez przejścia do nowej linii) moga powodować na systemie z SP2 problem. Jeżeli ktoś może sprawdzić to z chęcią przeczytam o tym w komentarzu. Drugi z problemów jest trochę poważniejszy - wygląda na to że w SP2 domyślnie włączona jest opcja użycia wsparcia sprzętowego karty sieciowej - Receive Side Scaling. Funkcją RSS (nie mylić z feed) jest między innymi wykorzystanie możliwości karty sieciowej do odciążenia systemu operacyjnego poprzez wykonanie pewnych czynności związanych z obsługą połączenia przez kartę. Problem polega na tym, że może to powodować błędy w systemach, które również zajmują sie przetwarzaniem połączeń, chociażby takich które realizują funkcję NAT itp. Opisuje to artykuł KB 927695. Jak donosi Thomas Sinder powoduje to również problem z działaniem ISA Server, co było do przewidzenia ze względu na fukncje ISA Server. Wniosek - przed instalacją SP2 na ISA Server warto przetestować taką konfigurację w laboratorium, w szczególności jeżeli nie możemy potwierdzić w dokumentacji czy nasza karta sieciowa wspiera Receive Side Scaling czy też nie. Możemy uniknąć niemiłej niespodzianki. […]

4. ISA 2006 testing Says:

   March 25th, 2007 at 2:44 am

   I am using SP 2 on windows 2003 R2 server, and the NIC drivers supports RSS.
   I have installed ISA Server 2006 Standard edtion (acting as a Edge firewall)
   no such issues have been faced by me????

   Will have to test with NA\T-ing on….

   will get back soon….

   Cool blog and site

   Cheers

   Monts

5. Thomas Shinder Says:

   March 25th, 2007 at 10:03 am

   Hi Monts,

   That’s good to hear. Let us know what happens.

   Thanks!
   Tom

6. Zen Says:

   March 26th, 2007 at 12:20 am

   OK, the issue is not across all ISA EE installations and seems to be restricted to the Broadcom NIC\’s. Intel NIC\’s do not have a problem. In order to get the ISA\’s RPC working again, simply disable RSS. Just had this problem on a Dell 1950 server. Took a week to diagnose. Wasn\’t happy, but then that is how it goes sometimes…..

7. Thomas Shinder Says:

   March 26th, 2007 at 7:58 am

   Doesn’t have anything to do with SE or EE — the EE problem is with ADAM.

   Tom

8. Doan Van Says:

   March 27th, 2007 at 12:05 am

   Hello, I’m using ISA 2004 SP2 EE on Server Window 2003 SP1 and one day I install SP2 for window 2003 SP1 and finish but open ISA to manage some but there’s nothing on it. when I right click on “Microsoft internet security and Accesleration Server 2004 SP2″ to Connect to configuration Storage server and connect to local host then I have Problem like that:

   “ISA server management was unable connect to configuration Storage server”
   Error 0×8007203a
   The server is not operation.

   so enybody help me
   thanks,

9. QuintaStation : Service Pack 2 para Windows 2003 Server en español Says:

   March 27th, 2007 at 2:18 am

   […] Service Pack 2 para Windows 2003 Server en español Desde ayer teneis listo para descargar el Service Pack 2 de 2003 server por fin en castellano: http://www.microsoft.com/downloads/details.aspx?di...d6d90a Oh, si alguno tiene un ISA montado que eche un ojo, por si acaso a: WARNING! Windows Server 2003 SP2 May Destroy Your ISA Firewall without Warning: http://blogs.isaserver.org/shinder/2007/03/23/warn...rning/ You cannot host TCP connections when Receive Side Scaling is enabled in Windows Server 2003 with Service Pack 2: http://support.microsoft.com/default.aspx?scid=kb;...927695   […]

10. Thomas Shinder Says:

    March 27th, 2007 at 5:41 am

    This is most likely the ADAM issue with SP2. Check my other blog entry on SP2 woes regarding this problem.

    Tom

11. Andrew English Says:

    March 27th, 2007 at 7:59 am

    The god father speaks!

    Great work Tom! This will certainly add a fuel to the fire on another forum where people think SP2 is the greatest thing to man kind.

    Andrew

12. Thomas Shinder Says:

    March 27th, 2007 at 8:04 am

    Hi Andrew,

    SP2 might be a good thing in general, though I have no idea. It’s definitely not a good thing yet for ISA Firewalls. We’re still trying to gather up all the SP2 bugs affecting ISA, so stay tuned!

    Tom

13. Dieters ISA blog : Windows Server 2003 Service Pack 2 und ISA Server Says:

    March 28th, 2007 at 4:21 am

    […] Dieter RauscherMVP ISA Server Share this post: email it! | bookmark it! | digg it! | live it! Published Wednesday, March 28, 2007 11:55 AM by rauscher Filed under ISA Server 2004, ISAServer 2006 […]

14. HerbyDumpling Says:

    March 30th, 2007 at 7:56 am

    We have just walked right into this RSS problem with a new installation of SBS 2003 on a Dell server with Broadcom NICs. Spotting the problem early on we tried using Q927695 but it did not appear to fix it. With reluctance we got on to Microsoft Support. They have stuck with the problem to resolve the issue. The new box is almost working great now.

15. Thomas Shinder Says:

    March 30th, 2007 at 8:59 am

    Has Microsoft come up with a solution yet? Or maybe Dell?

    Thanks!
    Tom

16. Stuart Says:

    April 2nd, 2007 at 4:25 pm

    THANK YOU - THANK YOU - THANK YOU - THANK YOU!

    Have been fighting with a new Dell fan-box with broadcom nic’s in all weekend. Died a horrible death upon the install of SP2 as above. Couldn’t find anything with regard to this untill the ISAserver.org email came through this afternoon and brightenend up my day!!

    http://support.microsoft.com/default.aspx?scid=kb;...927695

    That link worked a treat for us.

    Still waiting for Dell gold support team to phone me back though… been 14 hours so far… )

17. todd Says:

    April 3rd, 2007 at 8:51 am

    Herby:

    What did they do when MS stuck with the problem to resolve it? Is it s secret? Do tell!

    Todd

18. John Says:

    April 3rd, 2007 at 2:30 pm

    Beside editing the registry to disable RSS, is there another way?

    Someone mentioned that you can do that from the network properties. However, I could not find anywhere in the adapter properties to disable the RSS. Help please? Thank you.

    John

19. Eric Says:

    April 4th, 2007 at 8:28 am

    John,

    On a Dell PowerEdge 1950 with Broadcom BCM5708C NetXtreme II GigE NICs:
    1. In Device Manager, select the first Broadcom BCM5708C NetXtreme II GigE NIC
    2. Right-click and select Properties
    3. Click the Advanced tab
    4. Select “Receive Side Scaling” and change the default value from Enable to Disable
    5. Click OK
    6. Repeat for each other NIC

    Eric

20. Marcus Says:

    April 6th, 2007 at 11:57 pm

    Kudos to Eric!

    I too have a Dell PE 1950 (Broadcom NICs) running ISA 2006. After installing SP2 and losing all Secure NAT internet connectivity, I tried the reg edit with no success but changed the NIC settings as Eric suggested and VOILA. Luckily I experienced virtually no down time because of this site! Thanks Tom and company! Now, let’s see what else is broken……

    Marcus

21. John Says:

    April 7th, 2007 at 12:20 pm

    Hi Eric,

    Thanks for the suggestion. If I don’t see any RSS option within the Broadcom NetXtreme (not version II), does that mean my NICs does not support RSS and therefore I should not worry? I also viewed the property under our other NIC, which are INTEL 1000MT and also do not see such RSS option. We installed W2K3 R2 SP2 and ISA 2006 EE and it’s working flawlessly.

    So to understand correctly, so far the current problem with SP2 while running ISA 2006 is more toward Broadcom NIC that supports RSS?

22. Thomas Shinder Says:

    April 7th, 2007 at 12:45 pm

    I don’t think it’s just broadcom — I think any NIC that supports RSS will be affected.

    Tom

23. Thomas Shinder Says:

    April 7th, 2007 at 12:47 pm

    Other problems people have been having with SP2 on ISA 2006:

    https://blogs.technet.com/isablog/archive/2007/03/...mments

    HTH,
    Tom

24. reBoot Says:

    May 9th, 2007 at 1:00 am

    Great site, Tom!

    Just about to fresh-install a new box for ISA 2006 in a new domain. Will be publishing Exchange 2007 and SPS Ent.2007 and have a “pure split” DNS-setup. Should I still wait with SP2, or are the issues solved now (as it is over a month since the last post in this thread).

    The NIC is Intel Gigabit PCI-card (1 year old).

    Please link to new solutions if/when there are any.

    -JKK

25. Thomas Shinder Says:

    May 9th, 2007 at 4:13 am

    Hi JKK,

    Just make sure that RSS is disabled on your NICs, and you should be good for a clean ISA Firewall install.

26. Tyler Says:

    May 14th, 2007 at 4:55 pm

    Hi,
    I encountered the same problem with SP2. I am running a Dell Poweredge 1800 with an Intel Pro 1000MT NIC. I checked the device manager but don’t see any mention of it supporting RSS. After installing the W2k3 SP2 download it broke my ISA 2004 server. I ended up uninstalling SP2 to get internet back for the users. Any suggestions are greatly appreciated to help get the settings right.
    Thanks,
    Tyler

27. Jim Mulvey Says:

    May 23rd, 2007 at 3:40 pm

    As I understand it from the Microsoft article here: http://support.microsoft.com/?id=927695 the problem is that both SP2 and the NIC are trying to perform Receive Side Scaling.

    The documentation I’ve read from Microsoft here http://www.microsoft.com/whdc/device/network/NDIS_...S.mspx says, “Thus a software implementation of RSS could make the system perform worse than if RSS were not enabled. As a result, implementations should not support RSS if the network adapter cannot generate the hash result.”

    So, this suggests to me that the registry change which disables RSS at the software level would be the preferred approach and would ensure you still get the benefits of hardware RSS without the performance degradation of having your CPU handle the cryptographic hashing.

    Comments?

28. Thomas Shinder Says:

    May 24th, 2007 at 8:37 am

    Hi Jim,

    Probably, but hard to say for sure without actually testing it.

    Tom

29. the back room tech Windows 2003 SP2 problems on Windows 2003 SBS servers « Says:

    May 26th, 2007 at 9:31 am

    […] Before you install this service pack on a SBS 2003 server, read the official release notes. Then, read Susan Bradley’s unofficial release notes, which detail the proper way to apply the update. Update your NIC drivers, especially for Broadcom NICs prior to installation. Posted in troubleshooting, SBS, patches, upgrade, best practices, Windows. […]

30. Sam Says:

    May 31st, 2007 at 1:22 am

    Hi Tom ,

    I’m running ISA 2006 and smartfilter as content filtering which worked like a champ . Lately I’ve updated my server with win 2003 sp2 and I’m running into some strange networking problems .ISA stop network communications intermittently.I’m disabled the RSS registry key but the problem remain the same. Finally I’ve uninstalled win 2003 sp2 but the problem remain the same.I don’t know what changes has been made to my isa server even after uninstalling win 2003 sp2 .

    Please help !!!

    Sam

31. Neil Pearson Says:

    June 19th, 2007 at 7:18 am

    Hi All,

    I have installed SP2 on an ISA 2004 with SP3. I have had no problems so far.

    Rgds,

    Neil

32. Rhodzer Says:

    July 25th, 2007 at 6:25 am

    Had failed teaming after update to SP2 with Intel pro 1000 MT dual card. Had to install the latest pro2kxp utility, then it still did not work. tried uninstalling each card individually - no joy.

    Have then uninstalled the software (deleting the network connections) logged into the server at it’s physical location, re-scanned for hardware, picked up the two NIC’s. THen installed the pro2kxp software utility and magically everything is now working and the teaming options are available through comp man.

    cheers,
    Rhodzer.

33. abhinaw Says:

    December 3rd, 2007 at 9:44 am

    we faced this problem of RSS( TOE, Offload etc ) with Windows 2003 Sp2 and old BroadCom NIC. we updated the NIC to the latest one with RSS support. Broadcom BCM5708C Nextreme GigE. Driver date 4/4/2006, version 2.6.14.0.
    We have RSS disabled with registry changes and now we are getting error as published by microsoft in this article —–http://support.microsoft.com/kb/910904—–

    “The average call duration has exceeded 10 minutes. If this is not the expected behavior, please see article 910904 in the Microsoft Knowledge Base at http://support.microsoft.com for details on how to use the COM+ AutoDump feature to automatically generate dump files and/or terminate the process if the problem occurs again.”

    Appreciate any feed back on this issue.
    thanks
    -abhinaw

34. Michaelm Says:

    January 30th, 2008 at 5:24 pm

    Thanks a million for Eric’s solution. Worked a charmed. Saved a disaster, having the server back up with 30 mins.

    Thanks again for your invaluable post.

35. Jake16 Says:

    March 12th, 2008 at 9:01 pm

    Hi,

    Any update on this?

    Thanks

Leave a Reply