Which Security Event Log audit categories are most useful on a Windows client?

Let’s say you’re looking to maximize the value of the data logged to the Security Event Log on a huge number of Windows clients (say, Windows XP SP2).

Further, let’s assume that you’re not inspecting such logs on a regular basis, but instead you just want to keep the most critical events in case you have to track down some “suspicious activity” later.  [Suspicious activity would probably include such things as successful intrusions into the PC (whether by attackers or malware), which is going to be a losing battle but worth trying.]

You have two different sets of knobs to twiddle: which categories of security events will be logged, and how the security Event Log will be configured.  The categories are the more involved thinking, so let’s start with the Event Log configuration first, shall we?

Security Event Log configuration

The default Security Event Log size on Windows XP is a paltry 512 KB.  [It got boosted on Windows Vista, so don’t go yelling at Microsoft — they heard ya already.]  The question isn’t if you should increase its size, but by how much?

When it comes down to a “best practice”, I’ve always found it to be an arbitrary choice.  This choice should be informed by the level of activity you expect (or tend) to see — many customers who turn on all logging options can fill up a 10 MB log in the space of a week, but those who make more judicious choices can survive on 2048 KB for sometimes a month.

The upper limit is somewhere in the neighbourhood of 300 MB, but that limit includes all Event Logs (even custom event logs created by other applications, I believe) — this is documented in Chapter 6 of Threats and Countermeasures.  So for example, if you’ve already set the System and Application logs to 50 MB apiece, I would strongly advise a Maximum log size of somewhere around 150-200 MB for the Security event log.  [Note: there is a bug that causes problems with Security event logs over 50 MB, which hopefully has not only been fixed in Windows Server 2003 but also Windows XP SP2.]

Aside: I’m sure there are others you might find, but on my own Windows XP box I’ve got four additional custom Event Logs:

  • Microsoft Office Diagnostics
  • Microsoft Office Sessions
  • Virtual Server
  • Windows PowerShell

The next setting to consider is how the logs will respond when they (inevitably) fill up.  There’s a setting innocuously labelled When maximum log size is reached, and there’s no perfect selection for everyone.  I’ve generally advised people to choose Overwrite events as needed, since most times, my customers would be interested in having a record of the most recent activity on the PC (e.g. tracking down details of a recent virus outbreak or suspected break-in attempt).

Finally, if you’re really anal about your Security Event logs (and what security geek doesn’t ideally want to keep them around forever?), you can enable one or two other specialized settings created just for you — but should you?

  • WarningLevel: recent versions of Windows can warn the Administrator when the Security Event log is nearly full (the usual recommendation is 80 or 90% threshold).  Windows will record a single System event with EventID = 523.  However, this is really only useful in cases where the Administrator wants to archive all Security Event Log records for later analysis or compliance checking, and they don’t already have an infrastructure for collecting and centralizing this logging info.  Warning someone of imminent failure, when they have no way to avert disaster, is really just a tease.  Thus, the more useful setting is…
  • AutoBackupLogFiles: Rather than let the log files overwrite themselves, some would prefer to archive all log entries.  This registry setting enables Windows to automatically backup and empty the specified Event Log, so that all the entries are stored in a local file on disk.  This isn’t perfect (a malicious attacker could wipe them out, for instance) but in cases where you just can’t imagine copying the security Event log between the time the 90% alarm goes off and you get the time to deal with it, this can be an effective alternative.  The most significant consequence of this is, over time, you may end up filling the OS volume with these archived files.  However, shunting such saved data to a separate, non-OS volume — or monitoring for disk space — are the kinds of problems that aren’t difficult to solve.

Security Event Log Category choices

Now the tough part: deciding which Success & Failure event categories to enable.  Leaning on Eric Fitzgerald and Randy Franklin Smith, here’s the current thinking I’m advising my customer for keeping the noise down (and which you’re welcome to leverage, if our thinking seems to fit):

Account Logon

  • This’ll identify the local (i.e. SAM-based) usernames that users have attempted to logon at this PC
  • If you’re interested in tracking actual user activity and successful break-ins, then enable Success auditing.
  • If you’re interested in (and plan to actually investigate) attempted but failed break-ins, and if your users don’t use local accounts (and thus won’t be the overwhelming cause of failed account logon attempts due to fat-fingering their password), then enable Failure auditing.  Under such circumstances, this shouldn’t be a significant contributor to the security logs.
  • Recommendation: enable Success and Failure auditing.

Account Management

  • This’ll identify such things as account creation, password reset and group membership changes.
  • Under normal circumstances these should be highly useful records (both the successful changes and the attempts) — especially if you don’t often manipulate local accounts on your XP clients.
  • Recommendation: enable Success and Failure auditing.

Directory Service Access

  • pointless — this only applies to Domain Controllers
  • Recommendation: No Auditing

Logon events

  • In a non-domain context, this doesn’t add much value over and above Account Logon auditing
  • Recommendation: No Auditing

Object Access auditing

  • This is a tricky one.  It logs little or nothing by default, even when Success and Failure auditing are enabled for this.
  • Used correctly, you can collect information with a fairly high signal-to-noise ratio.
  • Used incorrectly, however (and I was as guilty of this as anyone in my early career, and am still guilty today), and you’ll wipe out any useful information that the security log might’ve otherwise kept for you.
  • For example, I’m currently recording “Handle Closed” and “Object Access Attempted” events dozens or hundreds of times an hour.  What is being accessed?  LSASS.  Why?  Because of a single “Everyone: Full Control” auditing entry I added to the EFS\Current Keys registry key, to try to track down some odd behaviour a few months ago.  I’d forgotten about this ever since, and now I’m filling my 10 MB security log every 36 hours.
  • If you follow a VERY specific set of SACLs as in the EricFitz article linked above, then you will get some real value out of this category.
  • Recommendation: only enable Success and Failure auditing if you have specific activity you’re looking for, but be VERY careful when setting any SACLs on the system.

Policy Change

  • I’ve never seen anything in this category that helps really track down malicious behaviour
  • While it may be interesting to highlight attempted (or successful) changes to Audit policy or assigned user rights, I’m extremely skeptical that any of this information would be conclusive.
  • However, with Windows XP SP2 and the use of Windows Firewall, there are a number of very specific audit records (e.g. Event IDs 851, 852, 860) that track changes in the Windows Firewall configuration.  [It’s unfortunate that there’s not better info on the source of those changes.]
  • If you’re using the Windows Firewall in XP SP2, these records could well be useful in isolating the source, cause, or spread of a malware outbreak.
  • Recommendation: enable Success and Failure auditing when using Windows Firewall.

Privilege Use auditing

  • One of the greatest sources of log pollution, with little practical application.
  • This looks very useful to a security geek on paper, but in practice 99% of the recorded events will be (a) legitimate behaviour and (b) completely harmless.
  • Recommendation: No Auditing

Process Tracking

  • Aka “Detailed Tracking” (which is how these events are labelled in the security Event Log)
  • A great way to swell the size of your security logs, unless your PCs run a very small number of applications for very long periods of time.
  • However, when you’re using Windows Firewall, Failure auditing will record (in Event ID 861) a number of potentially useful pieces of information about any application that attempts to open an exception in the Firewall rules.
  • This logging can be very frequent (I show over 2000 events in the last 36 hours on my PC), but will give very detailed information on the the Port opened, the process that bound it, and whether the process is a service or RPC application.
  • (One good non-security use for this auditing capability is to troubleshoot unknown application behaviours.)
  • Recommendation: enable Failure auditing when using Windows Firewall.

System events

  • The only semi-useful information I’ve ever found from this auditing are the startup and shutdown events, and they’re much more useful in determining uptime statistics (and otherwise unseen BSOD events) than they are for security.
  • Unfortunately, these events get buried under the amazing number of 514, 515 and 518 events that accumulate in the space of a few days.
  • Recommendation: No Auditing

Summary: Windows XP Security Event Log auditing category recommendations

Security Event Log Category

Recommended Audit Level

Account Logon Success, Failure
Account Management Success, Failure
Directory Services access No auditing
Logon events No auditing
Object Access auditing No auditing*
Policy Change No auditing*
Privilege Use auditing No auditing
Process Tracking No auditing*
System events No auditing

* except in unusual circumstances, see above.

Advanced Oddities

Per-user Auditing

  • As of Windows XP SP2, auditing can be enabled or disabled for any or all users
  • Each category can be separately configured as well
  • On a PC with many user accounts, this would be useful to help remove the less interesting entries
  • However, where few accounts exist, and for PCs not joined to a domain, per-user auditing is not advised

Windows Firewall auditing

  • As I hinted above, there are some aspects of Windows Firewall’s operations that can be logged to the Security Event Log, and which don’t get logged to the pFirewall.log.
  • For organizations using Windows Firewall, and especially those that don’t have a perfect idea of all the exceptions they need to open up on their user’s systems, this auditing can be extremely useful.
  • Recommendation: To capture this data, you should enable Policy Change (success and failure) and Process Tracking (failure) auditing on the target systems

File/Registry access auditing

  • If you’re interested in detecting attacks that tamper with system files, then EricFitz has some fascinating work you should examine
  • His work became the input for the Security Configuration Wizard in Windows Server 2003 SP1
  • Having had a quick look at it, there’s nothing that looks dangerous or unsuitable for an XP client
  • Recommendation: if you’d like a quick & dirty way to detect changes to system files, cut and paste those “file access auditing” settings from the SCW templates, and make sure that you’ve also enabled Object Access auditing (success and/or failure, depending on whether you’re after actual changes or just attempted changes)

Full Privilege Auditing

  • You can toggle a Registry setting known as (duh) FullPrivilegeAuditing, but be warned: these are default disabled for good reason
  • Recommendation: do NOT enable this setting

Audit the access of global system objects

  • Ever since this got added late in the NT4 service pack cycle, I’ve never quite figured out what this really tells me.  Eric doesn’t seem to interested in this either for most of us.
  • Recommendation: turn this setting Off

Audit the use of Backup and Restore privilege

  • This setting blows me away — it’ll fill up the most generous security event log, ’cause it creates an entry for each file that is backed up or restored
  • Recommendation: do NOT enable this setting

CrashOnAuditFail aka “Shut down system immediately if unable to log security audits”

  • Are you nuts?  Have you ever met a sysadmin that voluntarily puts in place a predictable Denial of Service attack?
  • If you’re that one-in-a-million organization that can actually implement this setting, I want to hear from you.  Yours is a tale I just gotta hear…
  • Recommendation: duh, do NOT enable this setting

For More Information…

Eric Fitzgerald is an old colleague of mine from my days at Microsoft, and I have an incredible amount of respect for the depth and persistence with which he pursued issues in the Auditing subsystem of Windows over the years.  He’s like the Rain Main of Windows security eventing, except I don’t think he’s much of a fan of Wapner. 😉  Eric’s “Windows Security Logging and Other Esoterica” blog is chock full of Windows security auditing goodness.

Windows Security Log Encyclopedia — Randy Franklin Smith’s take on Security Event Logs

Technet Events & Errors Message Center — detailed information backing up each security Event ID and what it means.

Deciphering Account Logon Events — in case you wonder what “Logon Type 5” really means…

Account Management — disabling the noise — and we’re done!

 

[Apologies to anyone monitoring my external blog, as this is a straight repost.  However, I’m assuming very few of you know about both, so I’m going to start reposting anything that’s applicable to both audiences.]

As David Hsing says: Best. Troll. Ever.

Holy crap that’s funny:

http://talkback.zdnet.com/5208-12355-0.html?forumID=1&threadID=31199&messageID=579806&start=43

Reproduced here for those (like me) too lazy to click through:

You are kidding aren’t you?  Are you saying that this linux can run on a computer without windows underneath it, at all?  As in, without a boot disk, without any drivers, and without any services?

That sounds preposterous to me.

If it were true (and I doubt it), then companies would be selling computers without a windows.  This clearly is not happening, so there must be some error in your calculations.  I hope you realise that windows is more than just Office?  It’s a whole system that runs the computer from start to finish, and that is a very difficult thing to achieve.  A lot of people don’t realise this.

Microsoft just spent $9 billion and many years to create Vista, so it does not sound reasonable that some new alternative could just snap into existence overnight like that.  It would take billions of dollars and a massive effort to achieve.  IBM tried, and spent a huge amount of money developing OS/2 but could never keep up with Windows.  Apple tried to create their own system for years, but finally gave up recently and moved to Intel and Microsoft.

It’s just not possible that a freeware like the Linux could be extended to the point where it runs the entire computer from start to finish, without using some of the more critical parts of windows.  Not possible.

I think you need to re-examine your assumptions. 

YES (for the sarcasm-impaired), this is a joke, and it’s NOT my writing.  Don’t bitch at me if you are rabidly anti-Windows — click on the link above and rant away to your heart’s content.

Debugging a Word 2003 runaway thread…but not successfully

I just experienced one of the usual “hangs” in Microsoft Word 2003 that happen pretty regularly when working on multiple, large documents for any significant length of time.  The WINWORD.EXE process is taking up 50% of my CPU (which as a dual-core processor, means that there’s a thread that’s somehow taking up 100% of the logical CPU for which it’s scheduled), and has been doing this for at least ten minutes now with no letup.

In my experience, these “runaway consumers of CPU cycles” just never quiesce — eventually I have to decide to kill WINWORD.EXE from Task Manager or Process Explorer, or else the offending process will consume that “CPU” from now until the end of time.

Maybe I was just bored today, ’cause rather than just kill the runaway process, I decided to see if I could dig a little deeper.  [I think Mark Russinovich has infected me with the idea that these are surmountable problems — though I wouldn’t dream of trying to make a favourable comparison between my haphazard hacking and Mark’s mad skillz.]

Process Explorer

Let’s have a look at a few screenshots, shall we?

image 
(Performance stats, in case that’s useful to anyone — though it doesn’t provide me any telling evidence)

image
(Listing of the threads currently instantiated in WINWORD.EXE including the main thread, which is the one causing all the problems)

image
(Stack contents for the WINWORD.EXE thread)

image
(Stack contents for GdiPlus.DLL thread, which was the only other thread with any activity under the “CSwitch Delta” heading)

Process Monitor

Once I decided to investigate, I fired up Process Monitor and limited it to WINWORD.EXE.  The activity logged is almost entirely like this:

image

Don’t strain your eyes too badly on this — I’ve included this just to note the incessant nature of the major activity here: a rapidly-repeating WriteFile operation on a single Temporary file (~WRS1954.tmp), interrupted once in a while by a smaller (Length of anywhere between 512 and 3072) ReadFile operation on the same file:

image

Interestingly, these ReadFile operations occur in an irregular but repeating pattern:

image

Also of note is the fact that this temporary file is constantly growing in size, and not just temporarily swelling the data stored within a pre-allocated file — I confirmed that by right-clicking on the Path in Process Monitor, chose “Jump to Location…” and simply Refreshed on the folder to observe the reported file Size was incrementing every time (over a span of 50 minutes, it grew by approx. 222 Kb, or 233657856 bytes).

If I look closer at the Handles for WINWORD.EXE, I notice that this is one of many Temporary files open by Word, which implies that the problem we’re experiencing is very specific to one type of unexpected activity (and not just affecting Word’s handling of Temporary files):
image
(Note: I intentionally “hashed” out the original filename, which is the last entry in the list pictured.)

One other piece of information: I tried to resize the Window in which the active document was being displayed.  Word appended “(Not Responding)” to its Title Bar, and that seems to have changed the behaviour profile of the WINWORD.EXE thread.  Since that point in time, Process Monitor did not record any further increase in the size of the ~WRS1954.tmp file, but recorded one additional ReadFile operation on the WINWORD.EXE file itself (Offset: 3998720, Length: 4096).  [WINWORD.EXE File version = 11.0.8169.0, Digital signature timestamp = May 31, 2007 12:38:03 PM]

Finally, I grabbed a full memory dump of the WINWORD.EXE process, using windbg.exe and the .dump /ma command.  I can’t say I know much about debugging a dump file, but I’ve got it on the off-chance that I ever find a good guide to debugging.

What Caused This?

Three circumstances I think contributed to this, though in my opinion none of them should lead to hung process (since I’ve done this more often without incident):

  1. I had opened a Word 2003 document directly from Outlook (it was attached to an email).
  2. The document had Track Changes enabled, and I’d already added Comments throughout the document.
  3. In the Comment I was just editing, it had scrolled off screen…
    image
    …and I had just attempted to apply formatting (I’d typed [Ctrl]-B and [Ctrl]-I rapidly,to bold and italicize) to a single word in the Track Changes panel below the document (the one that opens automatically when you keep typing in Comments that have already “scrolled off screen”).
     image
    (Note: I intentionally redacted the confidential text — but it sure ain’t artistic)

Caveat: While my experience with Word over the years has taught me that heavy use and abuse of the Comments feature leads to instability, I’m still miffed that I’d lose the recent batch of edits just because I’d foolishly tried to emphasize my point using basic formatting in a Comment.

So What Can We Conclude So Far?

I don’t know much about reading a stack trace, so this is all guesstimation on my part (plus a little intelligence gathered from a good Russinovich article).  The WINWORD stack indicates that Word has called ntkrnlpa.exe aka the Windows kernel.  It looks like it’s basically stalled (KiDispatchInterrupt) while creating a new thread (KiThreadStartup).  Looking lower in the stack, the first caller in WINWORD is labelled only “0x1a772b” — whatever that is, it’s beyond my skills to unearth the identity of that API.

The next one down in the stack, however, is wdGetApplicationObject().  There’s no information in MSDN that references this function, though a few pages on the ‘net do allude to it (mostly in the same kinds of searches I made).  The best info I could find was here, which I’m guessing is Word’s way of getting a handle to the overall Word “application object”.  However, without any further context, it’s very hard to imagine what is really going on here.

Turning to the GdiPlus stack, it looks like another kernel call that’s stalled (many similar references to “WaitForMultipleObjects” functions), all boiling down to a call to the GdipCreateSolidFill() API.  From what MSDN documents, this seems like a pretty innocuous function, having nothing to do with temporary files, only to do with UI.  I can understand this — by the time I’d looked at the GdiPlus stack, I believe the UI had “hung” (aka it was non-responsive).  So while this thread was also active, it’s almost impossible for it to be involved in this issue.

Then the only thing I know for sure is the temp file was growing due to some runaway operation, and the runaway operation (which was probably related to an attempt to format Comment text) at some point obtained a handle to the Word application object.

I’m guessing that the only way to get any closer to the root cause would be to dig into the memory dump.  And…bugger me, the dump I grabbed ended up with this as its STACK_TEXT (from !analyze -v):

0adaffc8 7c9507a8 00000005 00000004 00000001 ntdll!DbgBreakPoint
0adafff4 00000000 00000000 00000000 00000000 ntdll!DbgUiRemoteBreakin+0x2d

Guess that’s “the wall” for me.

"Go/No-Go" Decisions on MyPicasaPictures, Attensa Sync to Google Reader, W2MW++

I’ve done a lot of investigatory work in the last few lunar cycles of different development projects I’d dreamed up.  However, after the investigation and the internal decision making was completed, I didn’t do a good job of “closing the loop” with any of you who have wondered “what happened all those projects?”.

I haven’t thought much about the royal “you” in this work — I’ve been sharing the steps and findings, but recently I started to wonder what people would think in the future if they happened to search on something that led them to one of these projects’ articles.  I’d feel pretty frustrated trying to find out where these things led (if anywhere) and where (if anywhere) there might be code that came out of these efforts.

Well then, it’s time to close the loop on these — at least, as much as I am “decided” on any of these so far.  That said, I’m never committed to any particular decision like this if any new evidence surfaces in the future to challenge my assumptions.  So if anyone is interested in picking up where I left off on any of this, drop me a line to let me know (I’m always interested in this kind of experimental work), and if you’d like to bounce some ideas off me, or see if I’d be interested in participating, I’ll always be open to such inquiries.

MyPicasaPictures: No-Go

Bottom line: while the effort to understand the VMC development environment was instructional and probably honed my ability to figure out what to look for in future explorations, my overall impression of MCML is that it’s just too damned hard for amount of value I might derive from it.

That, plus the chronic and unresolved stability issues I’m seeing with Vista Media Center (exacerbated by the merciless complaints and teasing I receive from my wife, who keeps saying “The XP box was much more stable, wasn’t it?”) have pretty much convinced me to pave the box, downgrade to Windows XP and to give Beyond TV a try.  [Their SDKs and more open, flexible architecture look like the perfect place to invest .NET development efforts, and the customer satisfaction with Beyond TV seems far superior to Windows Media Center, at least based on my initial research.]

Attensa Sync to Google Reader: No-Go

I had already decided to move from Attensa for Outlook to NewsGator Inbox, and then a few weeks ago NewsGator announced that their previously $30 Outlook client would henceforth be available for FREE to any and all concerned.

While there was no conversion possible from Attensa to NewsGator (well, I could import the OPML, but I couldn’t sync the “read/unread” status of all my articles, nor transparently migrate old articles to the new folder structure), everything else about this has been a positive experience.  I’m totally addicted to the NewsGator Mobile Reader for iPhone, and the fact that it syncs with my Outlook “read/unread” status is just awesome.  Congrats, NewsGator!

Attensa, I wish you luck in trying to survive the competitive pressures from NewsGator.  If I didn’t know better, I’d guess this is the beginning of the decline for Attensa, even though I think their Outlook client is superior to the current NewsGator Inbox offering.

W2MW++: Undecided

When I first read about the “export to MediaWiki” capability in OpenOffice Writer 2.3, I quickly concluded that any work I or the rest of the community had done for an Office add-in would become a moot point.  [Amusing but not-entirely-inaccurate Spoonerism: my wife knew a guy who insisted that the term was “a mute point”.]

However, after using Writer 2.3 to convert a few relatively simple Word 2003 documents to MediaWiki format, I realize that they still have a long way to go to preserve real fidelity of layout and formatting in Word documents.  I have faith that they’ll get there, and that eventually Writer’s integrated engine will become the translation engine for .DOC & .DOCX, but I now feel like there’s a significant unmet need that the work I’ve invested so far in W2MW++ could still address, and that that unmet need will exist for quite a while yet.

That said, there’s one thing that’s been bugging me for a few months now: the name.  WordToMediaWikiPlusPlus is a clever extension of the Word2MediaWikiPlus project, and it makes obvious the heritage of W2MW++, but it makes it sound like the project is more “hardcore” than it really is.  If I had my druthers, I’d rename the project “Word2MediaWiki.NET” (W2MW.NET), to make it clearer that the project is based in .NET code, not C++.  I’d hate to think anyone would be disappointed by the fact that it’s written in one of these “shiny new” languages — there’s something more “honest” or “obvious” about using the “.NET” suffix instead.

Now all I have to do is figure out how to Rename all the dozens of “++” references throughout the project AND figure out how to get a CodePlex project renamed.  [THAT should be fun :)]

Windows Update — talk about shooting yourself in both feet…

Microsoft update - nonsense

For the love of Pete (who’s Pete you ask?  It’s a joke, son), who’s keeping watch over Microsoft customers’ safety and security?  For well over a year now, I’ve encountered Windows XP SP2 PC after PC, dutifully configured to automatically download and install all high-priority updates.  Some of these PCs, I’ve mothered over multiple times, hoping that I was seeing just a one-time problem that would be magically resolved the next time I arrived.

Microsoft even makes a big deal in its advertising about the fact that Windows Update (or Microsoft Update, if you’ve opted-in to this long-overdue expansion of updates across many Microsoft consumer and business products) “…helps keep your PC running smoothly — automatically”.  [And if you don’t believe me, check it out for yourself.]

Hogwash, I say.

Windows Update?  It’s more like “Rarely Update”, or “Windows Downtime”.

In almost every single case (and I suspect the rare PCs that weren’t this way, had been similarly mothered by some other poor lackey of the Beast from Redmond), I’ve found that I had to visit the Windows Update web site, download yet another update to the “Windows Genuine Validation” ActiveX control, install this piece o’ quicksand, and then subject my friend’s (or family member’s) PC to the agony of between one and three (depending on how long it’d been since I last visited) sessions of downloading and installing the very updates that they (and I) continued to falsely believe were being downloaded “automatically”.

In those cases where it’d been a year or more since the last occasion of hand-holding by me, the cycle of abuse wasn’t complete with a single session — I had to reboot after all “available” updates were installed, and re-visit Windows Update to find yet *another* batch of updates that magically appeared on this subsequent go-around.

How does this happen?  How could a service that is supposed to minimize the occurrence of unpatched PCs turn against itself so horribly?

I have to imagine that the WU (Windows Update) team doesn’t have any oversight or centralized control over the content that’s being hosted on their site.  If they did (and assuming they’re the folks who paid for the above ad), then they’d take their responsibilities more seriously, and make sure their site could deliver on the promise being advertised.

As it stands, it appears that the team responsible for Windows Genuine Validation feels it’s more important to ensure that their software is being explicitly installed by the end user, than to ensure that Microsoft’s customers are being adequately protected from the constant onslaught of Windows-targeting malware.

Each and every time I have visited the Windows/Microsoft Update site on these “under-managed” PCs (i.e. PCs owned by those folks who have left their PCs alone, as they’ve been promised to be able to by Microsoft), I’ve found that I had to perform the “Custom” scan, then accept the only-via-the-web download for the Windows Genuine Validation software, and only then is the computer capable of automatically downloading the remaining few dozen updates that have been queued up while the PC has been prevented by the requirement to download the validation control.

It seems like the Windows Genuine Validation team isn’t satisfied with their software getting onto every Windows PC in existence; they also seem bound & bent to ensure that every user is explicitly aware that they’re being surveilled by the Microsoft “licensing police”.

Why is it that Windows Update (or Microsoft Update) can update every other piece of software on my Windows PC automatically, but the license police can’t (or won’t) get its act together and make their (unwanted but unavoidable) software available automatically as well?  And don’t tell me it’s a “privacy” thing, or that it wasn’t explicitly allowed in the Windows XP SP2 EULA.  We’ve had plenty of opportunities to acknowledge updated privacy notifications or EULA addenda (hell, there’s at least one of those to acknowledge every year via WU, it seems), so that don’t fly.

So here’s my proposition: I’d love to see the Windows Genuine Validation team fall in line with the rest of the Microsoft “internal ecosystem” and figure out a way to make it so that WU/MU automatic updates actually become automatic again.  Wouldn’t it be grand if Windows systems around the world were still able to keep on top of all the emerging threats on behalf of all those individuals who’ve filled Microsoft’s coffers over the years?

Let’s get the current WGA control packaged up like any other High-Priority update and pushed down on the next Patch Tuesday (pitch it as if it’s similar to the monthly malware scanning tool).  If you have to, add in one of those EULA addenda (with or without a prominent privacy notification up front), and if you’re really worried, run a big press “push” that gets the word out that a privacy notification is coming.  C’mon Microsoft!  You’ve conquered bigger engineering problems before.  This one (at least to my naive viewpoint) can’t possibly be that hard…

MyPicasaPictures Part 5: Hacking the XP development environment for Vista Media Center applications

So I’m going through the Windows Media Center Application Step by Step guide (which incidentally, is in the XPS format — Microsoft’s pretender to the PDF throne, and a risky choice for any application to choose ahead of the widespread adoption of the underlying platform).  I’ve gotten as far as to add the References to the MediaCenter assemblies, when I realize (for the first time) that developing a Media Center application might very well mean that I have to have the development environment installed on a Media Center PC.

This seems like an odd dependency to have, considering all the other types of development projects that don’t necessarily require you to run your IDE on the target platform.  So I’m wondering if there’s a way to either (a) just drop the assembly files (i.e. the DLLs) into the expected directory and just reference them & go, or (b) I’ll have to figure out how to register those assemblies on my non-Media Center system.

On pages 5 and 6 of the Step by Step guide the instructions indicate to add references to the Microsoft.MediaCenter.dll and Microsoft.MediaCenter.UI.dll assemblies.  While it assumes you’re working from a Vista Premium/Ultimate machine and have those assemblies in the %SYSTEMROOT%\ehome folder, I found that copying those files from the VMC box and browsing to their copied location on my Windows XP SP2 system seemed to resolve the references just fine.  [I’m not convinced there won’t be other issues later on, but it’s at least worth a try.]

What’s The Sample Code Doing?

The code going into Application.cs looks interesting, but without any Comments to help us understand the purpose of each line, it’s not very instructive.  For me, this is all I’m able to infer:

  • we’re creating an Application class (though I don’t know what VMC developers would think of as an “application” — an add-in?  a piece of UI?  an assembly?  the whole Media Center process and children?)
  • we’re creating three Properties, both a private instance and its public instance (though I don’t really have any specific idea what these properties are meant to do)
  • the Application “object” is overloaded, but I’m not even sure if it’s a Property or something else
  • there are references to this, but I don’t really know what the “this” is referencing — is it the Application class, or is it something else?  [It doesn’t help that I’m not an expert at coding, so I don’t intuitively recognize what each of these things are, but it sure would help to reinforce my tentative steps into this if I had Comments that told me what I was looking at.]

I’m also puzzled by why this code doesn’t follow the Design Guidelines for Managed Class Developers, when nearly all managed code published by Microsoft adheres to these standards.  For example,

public void DialogTest(string strClickedText)

does not need “str” to prefix the ClickedText variable (aka Hungarian notation) — Visual Studio will always indicate the datatype with tooltips.

As another example, the private member variables

            int timeout = 5;
            bool modal = true;
            string caption = Resources.DialogCaption;

are all named using Camel case but without any prefix (such as a leading “_” character).  I understand that the Design Guidelines don’t specifically call out prefixing rules for member (? or private ?) variables, but apparently the guidelines do recommend the use of “this.” to prefix instance variables (which is awful confusing, as you can see above).  Personally I’d prefer to see “_” or “m_” prefixes on private member variables, but I’d at least like to see *some* consistency in the use (or non-use) of prefixes on variables.

While prefixing is a religious fight, the use of Hungarian is not — it’s clearly not recommended.

Compile Errors

I followed the Step by Step through to Build the Solution, at which point I got back one error in the compiler, at Line 55 of my code:

‘MyProject.Resources’ does not contain a definition for ‘DialogCaption’

If I followed the code adequately, the Step by Step has a bug (or at least an ambiguity) in the step Add Resources.resx/Add Strings.  Whereas the Step by Step guide actually wanted me to create one String called “DialogCaption” in the Resources file, I understood it to instruct me to create two Strings — one called “Name” and the other called “Value”:
Snippy0002

It seems pretty obvious now, but at the time my interpretation seemed intuitive.

 

After resolving that more trivial issue, the next Build confronted me with this error:

The command “%windir%\eHome\McmlVerifier.exe -verbose -assemblyredirect:”C:\VS2005 Projects\MyProject\MyProject\bin\Debug” -directory:”C:\VS2005 Projects\MyProject\MyProject\Markup”” exited with code 9009.

This turned out to be harder (and dumber) to diagnose.  There were no specific errors related to McmlVerifier and 9009, and the first real lead from Google referred to path quoting issues.  I finally realized that the McmlVerifier.exe file didn’t even exist on my XP (development) system.  Strangely though, it didn’t exist on the VMC system either – is this one of the SDK tools?  Yes.  However, for some reason the SDK didn’t install it into the %windir%\ehome directory.

Once I copied McmlVerifier.exe to the ehome directory, the Build completed successfully.

Strangely, most of the steps under Enable UI Testing and MCML Verification (steps 7-17) were redundant – they were already completed on my behalf.

Debugging

When I tried Debugging the solution, I got yet another error:
image

System.DllNotFoundException was unhandled

Unable to load DLL “EhUI.dll”: The specified module could not be found.
(Exception from HRESULT: 0x8007007E)

However, this time when I merely copied EhUI.dll to the %windir%\ehome directory, the Debugger threw another exception, slightly more impenetrable:
 image

System.DllNotFoundException was unhandled

Unable to load DLL “EhUI.dll”: The specified procedure could not be found.
(Exception from HRESULT: 0x8007007F)

The stack trace for this was:

at Microsoft.MediaCenter.Interop.RenderApi.SpWrapBufferProc(ProcessBufferProc pfnProcessBufferProc, IntPtr* ppNativeProc)\r\n
at Microsoft.MediaCenter.Interop.RenderApi.InitArgs..ctor(MessageCookieLayout layout, ContextID idContextNew, ProcessBufferProc pfnProcessBufferProc)\r\n
at Microsoft.MediaCenter.UI.MessagingSession..ctor(UiSession parentSession, ContextID idLocalContext, TimeoutHandler handlerTimeout, UInt32 nTimeoutSec)\r\n
at Microsoft.MediaCenter.UI.UiSession..ctor(ContextID idLocalContext, IServiceProvider parentProvider, RenderingInfo renderingInfo, EventHandler rendererConnectedCallback, TimeoutHandler handlerTimeout, UInt32 nTimeoutSec)\r\n
at Microsoft.MediaCenter.UI.UiSession..ctor(RenderingInfo renderingInfo)\r\n
at Microsoft.MediaCenter.Tools.StandAlone.Startup(String[] args)\r\n
at Microsoft.MediaCenter.Tools.StandAlone.Startup()\r\n
at Microsoft.MediaCenter.Tools.McmlPad.Main(String[] args)

Given that I’ve got the Vista version of EHUI.DLL in the right place, I assumed I’d have to “register” it (I forget the equivalent in the .NET Framework world) so that its procedures could be “found”.  However, before going down a “.NET theory” path I decided to google the error message.  The first five or so forum posts that came back pointed finally clued me in that in fact this may represent a second-order dependency in another missing DLL.  With that, I decided to just copy in the entire contents of my VMC’s %WINDIR%\eHome directory and retry debugging.

Debugging exponentially: Process Monitor, Dependency Walker

That wasn’t any more successful, so next I fired up Process Monitor and watched for any “NOT FOUND” errors that followed the EHUI.DLL load event (hoping that this was merely a case of a missing filesystem reference, and not something more sinister).  That helped me discover the following errors (presumably, as is often the case in these situations, mostly a list of false negatives):

  • McmlPad.exe called RegCreateKey() for HKLM\Software\Microsoft\Fusion\GACChangeNotification\Default
  • McmlPad.exe failed to find OLE32.DLL under C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727, and oddly didn’t try searching the environment’s PATH locations
  • devenv.exe searched multiple times for C:\temp\symbols.flat.txt and McmlPad.pdb (symbols) under C:\temp\symbols\McmlPad.pdb\6B1042D64F29479FA1C07939AE072D941\, C:\Windows\symbols\exe, C:\Windows\exe, C:\Windows
  • McmlPad.exe failed to find C:\WINDOWS\assembly\GAC\PublisherPolicy.tme (and didn’t look anywhere else)
  • McmlPad.exe tried looking for a couple of files in the GAC search paths before it went looking directly for the file in C:\Windows\ehome:
    • Microsoft.MediaCenter\6.0.6000.0__31bf3856ad364e35
    • Microsoft.MediaCenter.UI\6.0.6000.0__31bf3856ad364e35
  • Once McmlPad.exe successfully loaded EHUI.DLL from C:\Windows\ehome, it (successfully) opened these files:
    • C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\sorttbls.nlp
    • C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\sortkey.nlp
  • Then devenv.exe successfully loaded C:\Program Files\Microsoft Visual Studio 8\Common7\Packages\Debugger\cscompee.dll
  • Then McmlPad.exe loaded C:\WINDOWS\assembly\NativeImages_v2.0.50727_32\System.Configuration\eee9b48577689e92db5a7b5c5de98d9b\System.Configuration.ni.dll

Hmmm, I’m not sure I learned much from that.  It looks like the application being debugged (McmlPad.exe) was looking for some GAC registration info and a few obscure files.  However, it’s likely that this is expected behaviour even on a working system.

So I went back to my google search results which convinced me to try DEPENDS.EXE to see what it would say.  I expected nothing, frankly, but that gave me two new leads I wouldn’t have otherwise found: it indicates that EHUI.DLL is looking for DWMAPI.DLL and DXGI.DLL, neither of which were present anywhere on my system.

Fascinating – one of the first Forum posts I found referencing DWMAPI.DLL indicates this file may not be needed when “linked” by a multi-OS-targeting application.  However, I suspect that for VMC libraries like EHUI.DLL, these two missing DLLs are not “load on demand” libraries – they’re just libraries that don’t appear on non-Vista platforms.

Once I grabbed copies of these two files from my Vista machine and dropped them into the %windir%\ehome folder, DEPENDS.EXE warned me that D3D10.DLL and NUCLEUS.DLL were required for DXGI.DLL, and that EHTRACE.DLL (a demand-load library) was missing.  Okey-dokey… and then I’m “warned” that even with all these files in place…:

Error: At least one module has an unresolved import due to a missing export function in an implicitly dependent module.
Warning: At least one module has an unresolved import due to a missing export function in a delay-load dependent module.

Feeling lucky despite all evidence to the contrary, I re-ran MyProject from Visual Studio, but no love was had – still and all, “the specified procedure could not be found”.  OK, let’s continue…

The unresolved imports were flagged in the following libraries on my XP development PC:

Library

Unresolved Import(s)

MSVCRT.DLL _except_handler4_common
_ftol2
_ftol2_sse
D3D9.DLL Direct3DCreate9Ex
USER32.DLL IsThreadDesktopComposited
ADVAPI32.DLL RegGetValueW
MPR.DLL WNetRestoreConnectionA

So, the obvious question is: if I copy *these* libraries from my VMC computer to C:\windows\ehome on my development XP computer, will that suffice to allow McmlPad.exe + Visual Studio 2005 to successfully debug my VMC app?

And the answer is: not quite yet.  Unfortunately, when loading EHUI.DLL, Windows will still end up loading the standard libraries (such as MSVCRT.DLL) from the PATH (i.e. %windir%\system32).  I realize that Windows is just trying to behave nicely, not being “tricked” into loading rogue versions of already-installed libraries, so I am not too upset by this.  However, this has turned into a much more difficult problem than I first anticipated.

Files copied to my development system so far:

  • %windir%\ehome\Microsoft.MediaCenter.dll (from VMC system)
  • %windir%\ehome\Microsoft.MediaCenter.UI.dll (from VMC system)
  • %windir%\ehome\ehui.dll (from VMC system)
  • %programfiles%\Microsoft SDKs\Windows Media Center\v5.0\Tools\McmlVerifier.exe (to %windir%\ehome)
  • %programfiles%\Microsoft SDKs\Windows Media Center\v5.0\Tools\McmlPad.exe (to %windir%\ehome)
  • %windir%\system32\dwmapi.dll (from VMC system)
  • %windir%\system32\dxgi.dll (from VMC system)
  • %windir%\system32\d3d10.dll (from VMC system)
  • nucleus.dll (from I don’t remember where)
  • %windir%\system32\msvcrt.dll (from VMC system)
  • %windir%\system32\d3d9.dll (from VMC system)
  • %windir%\system32\user32.dll (from VMC system)
  • %windir%\system32\advapi32.dll (from VMC system)
  • %windir%\system32\mpr.dll (from VMC system)

  Tidbits

  • 😦 “I will say that if you are a hobbyist then MCML is not likely to be very friendly to you.”
  • 😦 “It took me a long while to create a gallery that displays a list of images from an ArrayListDataSet.”
  • 🙂 “I have persevered in the days since my last post and have managed to get a repeater/scroller working to emulate the behaviour of the ‘my music’ section of media center.”
  • 🙂http://mobilewares.spaces.live.com/ has a good tutorial on making the sliding menu at the top of the screen.”
  • 😦 “Following the rumour about Microsoft encouraging their internal teams to deprecate the use of an underscore prefix for private fields, I have decided to do the same.”  [I just got my head wrapped around the use of underscores for private fields, and now I need to unlearn that behaviour just as quickly.  At least I haven’t done too much damage with it.]
  • 🙂 “If you can’t easily work out where the variable is defined (local scope, local parameter, member variable) then your code is too complex and needs refactoring.  It’s that simple.”  [I can handle that idea, yeah.]

MyPicasaPictures Part 4: Smart Client Software Factory — a deeper look

Having determined that of all the Patterns and Practices guidance, the SCSF is likely the one with the most to offer my project, I decided to see what following the Getting Started steps would get me.

And just my luck, I can’t even get past the first steps — to create a Hello Word application, I need to be able to select from the “Guidance Packages Project” in Visual Studio, but it appears they didn’t even get installed.  I’ve run the SCSF (May 2007) installer twice and selected everything, but still it doesn’t add any entries to the New Project dialog.

What’s a little disturbing is that when I started to peruse the discussions to see if anyone else had run into this, the detailed steps that people had to go through to troubleshoot problems with this guidance were shocking (see this thread, for example).

I really didn’t expect this “factory” to be this complex — I was hoping for a library of code and simple steps to piece different bits together.  What I seem to have gotten instead is another “learning opportunity”, much like the MCML rabbit hole — some limited set of benefits that’s masked by a steep up-front learning curve, and that only benefits your use of that proprietary set of tools.

Does anyone ever just try to create a set of code snippets or object libraries?  Why does everything require learning a whole new development paradigm, and some new hierarchy of newly-overloaded terminology that doesn’t relate to any of the stuff upon which it purports to build?

Grrr…

Dependency Checker: decoupled from install?

Wow, so I decided to go right back to the beginning, and see if there were any hidden steps I missed the first couple of times.  It turns out that, without making too much of a big deal of it, the SCSF team has written a separate downloadable tool that checks for all the required and “optional” (?) dependencies [who ever heard of an optional dependency?  That’s a new one on me].  After all this time, I’ve come to expect that the installer for the program I’m after would check those dependencies as a first step, and would tell me to go get the pieces I’m missing if it detects that it won’t be able to work if it installs.

No, for whatever reason these guys seem to have decided that they want their end users to “educate” themselves on what it really means to use their guidance, so they seem to have intentionally left at least one land mine in place for most newbies to trip over.  I dunno, but if most .NET applications can test for the presence of the required .NET Framework (and halt the install if it’s missing), how much harder could it possibly be to add this dependency checker code as a pre-install step and direct users to the pieces they’re missing?

This reminds me of the days when you downloaded Eudora or Mosaic to “access the Internet”, and then later learned that you needed to download Trumpet Winsock or some other TCP/IP stack before they’d work.

YAFR (Yet Another Freakin Runtime): Guidance Automation Extensions

Well, so it looks like I’m missing something called the “Guidance Automation Extensions“, which sound reminiscent of the Office PIAs — yet another set of libraries that are necessary before the specialized development software will ever work.  Why does everyone seem to want to carve off a set of “foundation libraries”, rather than just include them in their software package itself?  Why do these scenarios inevitably require the end user to become an expert in the internals of the application, rather than just make it as easy as possible to get up and running in one step?

Maybe it’s the Micro-skeptic in me, seeing how often people at Microsoft fooled themselves into thinking that *their* little piece of the platform was the *most* important thing around, and why *shouldn’t* our users spend some time learning how our stuff is supposed to be used?  It’s just such ridiculous arrogance, and in a lot of ways, I suspect it stems from the hiring process itself — hire these geeks straight out of college (with no previous real-world experience or humility i.e. ever failed at anything in their lives), tell them they’re the absolute best of the best, and then set them loose on a project.  Of *course* they’re going to believe that whatever comes from their mind is the product of god.

Now, after installing the so-called “GAX”, re-running the SCSF installer enables me to select the “Automated Guidance” selection (which had been greyed out before –  which in other installers simply means that you’ve already installed those bits, and don’t need to worry about them again).

Why CAB in the first place?

However, this is just the beginning of the confusion – for a tool that’s supposed to make it easy to build a well-patterned application, the discussions and blogs around SCSF make it clear that this is no library – this is yet another “only once you’re expert with it should you use it” tool.

Bil Simser says, “The point is that SCSF made me more efficient in how I could leverage CAB, just like ReSharper makes me a more efficient developer when I do refactorings.”  However, what he pointedly fails to say is that SCSF makes him more efficient as a developer.  The debate assumes that you have already determined you must use CAB, and once you’ve fallen down that crevasse, what’s the best way to crawl back out?

I guess I need to take this one step further back, and understand the value of CAB to my current application development needs.  Until that time, I’m going to be very skeptical of having to learn the internal logic and complexity of yet another proprietary tool, and be very resistant to investing the time needed to learn enough to make it useful.

A little earlier in his blog, Bil Simser has this to say:

I’ll be the first to admit that CAB is complex. EntLib is large. There is a lot there. As Chris said this morning in what I think was an excellent response to the entire discussion, CAB for example is not just about building maintainable WinForm apps. I like CAB as it gives me a bunch of things and they all work together in a fairly harmonious way. EventBroker is a nice way to message between views and keeping the views separate; CommandHandlers allow me to hook up UI elements indirectly to code to execute them; the ActionCatalog let’s me security trim my commands (and in turn my UI); and the implementation of the MVP pattern using views lets me write presenter tests and keep my UI thin. This all makes me feel good. Did it take me a while to get here? Absolutely. I’ve spent the better part of a year learning CAB, EntLib, ObjectBuilder, WorkItems, and all that jargon but it’s no different than learning a dozen different 3rd party libraries. I simply chose the MS path because it was there and everything was in one neat package. If you packaged up Castle, NHibernate, StructureMap, and others together in a single package maybe I would have chosen that path (and is there really two different paths here? I use both tools together anyways).

Yipes!  Better part of a year?  [And that’s not even counting the SCSF learning curve!]

Not to mention that even the experts wouldn’t want to try to explain it in a short amount of time:

Unfortunately [Entlib/CAB is] not something I could introduce at a conference or User Group session and describe the entire stack in an hour, so I tend to avoid showing off applications and concepts using it as it just turns into a discussion of what (SmartPart) means instead of the main goal like describing MVP which I can do with my own code.

And the promise that even with all of what’s bundled into Entlib and CAB, you could still find yourself dragging in all sorts of other crap:

Eventually I could have a really ugly monster on my hands with copies of Castle, StructureMap, CAB, EntLib, NHibernate, log4net, and who knows what else all living (hopefully) together in happy existence. I don’t want that.

More words to give me pause, from Chris Holmes:

I’ll also say this about ObjectBuilder : I agree with Jeremy; I wouldn’t use it on its own. It’s hideous. It is overly complicated, undocumented and cumbersome. It is the worst part of EntLib and CAB. Fortunately, when using EntLib and CAB I don’t have to manipulate ObjectBuilder. If I did, I’d have abandon both tools a long time ago.

So we come to this question: If I don’t like ObjectBuilder, then why am I using CAB or EntLib?

Now, some people might say “that’s Big Design Up Front”. I’d argue otherwise. Our ultimate goal was testability. We wanted to adopt an MVP architecture so that we could have a very testable UI layer. We wanted automated tested; we wanted automated builds; we wanted the confidence that comes from having a vast suite of unit tests; we wanted the confidence to refactor our code without fear of breaking a tightly coupled system. So our goal was not to have “nifty tools”, but to have a framework to build upon that would give us our ultimate goal: a testable UI layer that we all would have confidence in.

My assumption was that [rolling my own code] would take a considerable amount of time (and not as BDUF; but as time spent building the pieces when necessary. The bottom line is, it still takes time, no matter how you allocate it, and I wasn’t certain how much). That assumption was probably unfortunately influenced by looking at the CAB as a frame of reference and saying, “Holy cow, there’s a lot there…”

So we went with CAB. The idea was that we thought it would help us get out of the gate faster. I thought it would take less time to grok the CAB and make use of it than roll my own code. Maybe that was an error in judgment. It’s certainly possible, I am human and very flawed. But I did manage to grok the CAB pretty fast compared to other adopters, so it seemed like a good decision to me at the time.

I chose CAB because I thought it was the best solution at the time, given everything I knew about software design and weighing all of the other factors involved. I might make a different decision today, given the amount of knowledge I’ve accumulated and the tools that have emerged. I might do just what Jeremy and Oren propose, and grab an IoC tool like Castle and grow my own solutions as necessary.

SCSF, CAB & Entlib: just not ready for the Media Center developer crowd

While having an IM conversation with one of my respected colleagues today, I complained about all the time I was investing in learning about yet another framework:

I just dove deep into the CAB/SCSF/MVC-P crevasse, and it’s amazing how much of learning the bit you need depends on learning the next-layer-down foundational piece – and how many f’n foundational layers there are once you start looking closely.

The thing that J.D. Meier & co did well was provide some automated “wizards” that setup all the skeleton code once you know which pattern you want to use.

The part they addressed poorly is to provide any “entry-level” patterns for weekend hackers like me – everything is all about “steep learning curve, big architecture, lots of pieces to re-use once you figure out the puzzle up front”.  I’m not even convinced I ever want to do this kind of project again, and the amount of setup to even try it is making it less likely I’ll even do it once.

It strikes me that if I were to adopt one of these mammoth frameworks for the MyPicasaPictures project, the only advantage to the pre-defined pieces that are available to fire up at will is that other people (already familiar with the same framework) would have an easier to finding a customizing the pieces they wanted to bolt into.

For the rest of the Media Center community, if they wanted to extend the application or plug in a new module, they’d be forced to suffer the same learning curve as I’m facing right now.  This is not a community that I’d expect to be familiar with these enterprise development frameworks, nor do I feel particularly cruel enough to try to foist such a hairball upon them.

Fascinating quote to end this research vein with:

The end result is that this [anti-Agile programming] architect and I could not possibly coexist on the same project.  My longstanding opinion that traditional architects are useless if not downright harmful remains completely intact.  I guess you wouldn’t be slightly surprised to find out this guy is a big proponent of software factories as well.  Of course, this guy had more people in his talks and better evaluations than me, so what do I know?

If his description is any indication, and if I assume the likelihood that factories-based development suits the Media Center developer crowd is slim to none, then let’s just forget about the whole idea of using factories/frameworks for VMC-oriented development.

Tidbits

 

Next stop: back to the drawing board.  Perhaps I’ll work on a few crisp use cases to anchor the development of one or more rounds of Agile-esque coding.

MyPicasaPictures Part 3: Abstraction, Encapsulation, Model-View pattern — starting on the design

…I’ve never actually implemented a design pattern before, so this’ll be a learning experience all to itself.

However, I have implemented in previous apps what I’ve understood as the distributed application n-tier logical separation approach.  That is, I’ve tried to implement a basic separation/abstraction between the data access code and the “display” (presentation layer) code.  I’ve also tried to encapsulate some of the more substitutable functionality into utility classes, or wrap the access to a particular application behind a generic function call, so that I (or someone else) could substitute an alternative approach or target later on.

With that in mind, here’s a few ideas would make sense for MyPicasaPictures (without an understanding of what the Model-View pattern recommends):

  • provide a class for accessing the remote web service (Picasa Web Albums at first, but others later on), which would provide methods like GetMetaData() & SetMetaData() [both overloaded for one or many Pictures], UploadPicture(), UploadPictures(), DeleteRemotePicture(), RenameRemotePicture(), and Authenticate().
  • providing a class for the local Pictures and Folders, if needed, to create any custom Properties or Methods that aren’t entirely suited to this web services work.  Perhaps there’s some class in the VMC SDK for managing Pictures that includes the specific Properties & metadata that’s exposed by VMC?
  • providing a class for tying together the boring stuff like Startup, Shutdown, cleaning up connections, releasing resources.

[Later…]

Funny how, once I reviewed the VMC SDK’s Model-View Separation discussion, it turned out to be pretty much what I’ve described above.  I was hoping for something more granular, more sophisticated — something I could really sink my teeth into to get a head-start on this web services stuff.

Maybe the…simplicity is because this is just a subset of the Model-View-Presenter pattern, which itself has evolved into more sophisticated Supervising Controller and Passive View patterns.

Model-View, and the VMC-specific code

I’ve been thinking about this for a few days, and I’ve been hammering pretty hard on the MediaCenterSandbox community (trying to get my head around the MCML development model — NOT an intuitive thing, let me tell you).  The more I’ve been digging into this, the more planning and learning I’ll need to do before diving into UI development.  That was the most obvious areas to start (or at least the area I usually start with, giving me more time to mull over how the underlying classes will be divvied up and various functions will be implemented).  Media Center UI work also seems to be where all the ‘talk’ is.

However, I finally had a revelation today, that’ll make my life easier and let me get down to building something: why not start at the bottom, and build my way up?  I was thinking about the Model-View pattern, and it seemed obvious all of a sudden, to first develop “…the logic of the application (the code and data)…”.  Why not generate the web services client code, the data send and receive code, then create the local proxy stubs that call the web services functions, and then the functions for accessing and manipulating the local Pictures and Folders?

While I stabilize the “model” code, I can continue to investigate the MCML UI option, as well as other ways that I might “surface” the functionality I’m building in the “model” portion of the code.  [What if I just built a normal WinForms app?  Or a command-line batch processor?]  [Or why not just fall back to Picasa and be done with it?]

And since the “model” code would all be native C#, no MCML or VMC-isms, there’d be no lost work — all good ol’ fashioned reusable lessons, plus getting to know the Google Data API while I’m at it.  Can’t really lose with this approach, now can I?

 

Tidbits

  • Apparently, Background Add-ins are *not* persistent processes — i.e. they’re not intended to run the whole time from startup of the Media Center UI until it’s closed, or the box is rebooted.  [God only knows what the hell they’re good for if they’re as ephemeral *and* invisible to the user…]
  • MCE Controller — a SourceForge C# project enables programmatic control of the MCE UI.  Last updated a couple of years ago, so it’s hard to know how much of this is still relevant to VMC, and whether much of it has been replaced by the Media Center class libraries.

MyPicasaPictures Part 2: Rolling up my sleeves, sketching out a Spec

I find that the more time I spend detailing exactly what features and behaviours I expect from something I’m building, the faster and more successfully I’m able to actually build the thing I’m seeking.  This is a truism of technology development (and an old boss would say, of project management), whether you’re a one-person shop or a team of thousands.  However, it’s a lesson each person has to learn and digest to be able to figure out exactly what works best for them.

mini “Functional Spec” for MyPicasaPictures

Here’s a list of the features I’ve dreamed up so far, and the priority (1=high, 3=low) I’m personally assigning to them:

  • Pri1: upload a currently-selected Picture to (the default, if such a thing exists) folder/album/library (or whatever Google calls its collections) on the photo-sharing server
  • Pri1: enumerate and select the working folder/album/library when there are multiple folders to choose from
  • Pri1: authenticate to remote server
    • Pri3: cache remote server’s credentials securely, using DPAPI
  • Pri2: view names and/or thumbnails of pictures already in the selected remote folder/album/library
    • Pri3: enumerate and present the other metadata associated with each picture (e.g. Tags) and each folder/album/library
  • Pri2: write the code so that it abstracts the service-specific logic – enabling future versions of this app to easily add support for other server-based Photo sharing sites such as Flickr, Windows Live Spaces, Facebook, etc.
  • Pri2: Assign a new tag for a photo that has already been uploaded
    • Pri3: enumerate existing tags from remote server, and allow user to assign tag(s) from that set (along with new tags not part of the existing enumeration, in a mix-and-match formation)
  • Pri2: collect tags from the user for photo about to be uploaded, and submit the tags simultaneously (if supported by the photo-sharing service) with the upload, or right afterwards (if simultaneous isn’t supported)
  • Pri2: upload a batch of photos all at once (i.e. entire contents of a local folder that has been explicitly selected by the user).  Note: this may not be possible, as buttons or other controls cannot be added to the existing VMC UI.
    • Pri3: pick & choose a set of photos to be uploaded (e.g. subset of a single local folder, subset of photos in > 1 folders)
  • Pri2: delete existing photos in online albums (one at a time)
    • Pri3: delete existing photos in online albums in a batch (e.g. a subset of one album, or a subset of photos that span multiple albums)
  • Pri3: resize the photo(s) before they’re uploaded
    • Pri3: enumerate and present “spinner”-based choices control for the user to select one of the photo sizes “supported” by the remote server (if there’s any sort of default/preferred sizes that the server chooses to assert)
  • Pri3: rename the to-be-posted picture where another of the same name already exists on the remote site

Questions that keep coming to mind

  1. How do I implement client-side support for a SOAP- or REST-based Web Service in MCML apps?
  2. What “model” should I aspire to use for this kind of development? [cf. Gang of Four]
  3. Where will the entry point(s) for this add-in be located?  Steven Harding crystallized a suspicion I had been gathering on this topic – “Unfortunately, you can’t add anything to an existing Media Center interface.  So there’s no “Send This Folder to Picasa/Flickr” possible.”
  4. What exactly is a Start Menu Strip, and why are only two of them allowed at one time (and why does VMC seem to “punt” the oldest one out unceremoniously when a third is added)?
  5. What is the real difference between a VMC application and a VMC “background application” (other than the obvious visibility issue)?  i.e. Under what circumstances would I want to use one approach and not the other?
  6. What does “running on the public platform” mean in terms of (a) additional functionality that’s possible, and (b) what kinds of security restrictions are lifted on apps running on that “public platform”?
  7. If the More Information (“i” button) is so strongly discouraged, how should we provide that kind of “added functionality” in context of the application, ONLY when the user wants it, and WITHOUT cluttering up the UI in a way that makes it harder for most users to get their basic needs met?
    • Should we use a horizontal, multi-layer menu that’s presented on the Recorded TV screen?
    • Should we try the vertical stack of buttons that stay resident for all contexts, such as you see when you browse the detailed info for a Movie?
  8. Why is it that 3rd parties can’t add controls to the VMC UI, but Microsoft gets to change the user experience whenever they please (e.g. with the “Internet TV (beta)” object that quietly inserted itself – without asking, and without giving me any visible way to opt out – on the TV menu strip a few months ago)?
  9. What parts of the VMC UI are off-limits to third-parties?
    • Adding new objects to existing Start Menu Strips?
    • Adding new objects to existing “More Information” context menus?
    • Adding new sorting/filtering options to existing collections of content e.g.
      • I’d love to add “Show only Movies” to the “By Title” and “By Date” choices currently enabled in the Recorded TV collection
      • I’d give my left…toe to be able to sort a TV show’s episodes by the “Originally Broadcast” date, so I could accumulate a bunch of episodes of some show and then watch them in the order they were meant to be seen, not in the order they happen to have been recorded
    • Adding new tiles to the “More Programs” collections that are buried one click away from the Start Menu?

Funny

  • 🙂 “I’ve watched a video on Channel 9 where Charlie Owen and a programmer (Mark Finocchio?) demonstrate how to do basic MCML.  That was slightly more illuminating (though it took 35 minutes to get to the programming), but then I discover that you basically have to create your own buttons from total scratch. It really is back to the ark stuff.”

Painful

  • 😦 “I’ll check in to the ability to catch the More Information button — last I recall the handler is there but didn’t work exactly as planned.”  [sounds like one of those famous understatements of the year…]
  • 😦 “There are only the very basic visual elements – ‘Graphic’, ‘Text’, ‘Colorfill’ and ‘Clip’. Everything complex – like a button, menu, scroller etc. must be built from those four visual elements.” [also see 😦 ]
  • 😦 “This question comes up a LOT.  Enough that I covered it in my blog… Please check out the article called ‘Scope in MCML’.”

Next Steps

Here’s the list of beginner articles I’ve seen multiple folks point newbies at, to get a feel for what I’m about to take on:

  1. VMC SDK‘s “Step-by-Step” walkthrough (linked from the Windows Start Menu once you install the v5.2 or v5.3 SDK)
  2. MCML: UI’s (Steven Harding)
  3. UI Properties: Making UI’s Flexible (Steven Harding)
  4. Model-View Separation (Steven Harding)
  5. Stage 1: A Basic Layout (Steven Harding) through Stage 11
  6. GData .NET client library “Getting Started” guide
  7. cURL client for testing upload of files to Picasa

Source code to investigate: (as it might have some hooks already worked out that I won’t have to learn from scratch)

Picasa Web Albums Data API + Vista Media Center = MyPicasaPictures

I’ve found over and over, if I don’t upload pictures to the web when I first download them from my camera, then there’s little or no chance I’ll do it later.  Out of sight, out of mind.

But I wonder…

If I had an add-in command in the My Pictures sub-app of the Vista Media Center, would I be more inclined to upload pictures to the web then (and give my mom at least a glimpse into my life more than once a year)?

Google developer help: plenty

Since I’m already personally using Picasa’s companion Web Albums for what few pictures I’m sharing, I’m inclined to tie the online component of such an add-on to Google.

Having already investigated the Google APIs, I figured there was likely an API tuned for the Picasa Web Albums, and so there is.  Further, it appears that it provides the ability to post photos, create albums, update & delete photos, manipulate tags, and review existing albums/photos/comments/tags.  Sounds like it provides more than I was even looking for.

From what I gather with a quick skim, the basis for the “PWA” Data API is the Google Data API.  And while I don’t know whether it’s absolutely necessary, Google provides a .NET Client Library (currently v1.1.2) to speed web services client development.

Similar apps

Big Screen Photos v2 — the Yahoo Flickr client for Vista Media Center.  It doesn’t enable you to upload the pictures hosted in Media Center, but rather to view the pictures that are already available on the Flickr site.

Yougle Vista — provides access to online Video, Music and Picture sites including Flickr.  Still no upload capability.

PhotoConnect — extends My Pictures with some interesting functionality including the ability to email pictures to others.  However, it looks like this vanished in a puff of smoke and synthetic ash.

FlickrMCE — another viewing app, only for Windows XP MCE 2005.

PictureBook 1.0 — screensaver add-in, not really all that similar after all.

Windows Vista Media Center add-ons: sorting through all this new technology

Most add-ons for Windows Media Center 2005 (i.e. Windows XP) or earlier were written as “hosted HTML” apps that ran outside the Media Center shell.  Now, while hosted HTML apps may still work in Vista, the drive is towards either the MCPL/MCML or XBAP/XAML:

  • MCPL: Media Center Presentation Layer (the programming model)
  • MCML: Media Center Markup Language (the managed code language derivative)
  • XBAP: WinFX XAML Browser Application (the programming model)
  • XAML: eXtensible Application Markup Language (the language in which XBAP is more-or-less implemented)

However, it appears that the XBAP/XAML model has already been dumped by the eHome team.  Lovely, not even a year out of the gate and they already drop support for the “future-forward looking” programming model — I wonder what hurdles they had to clear to be able to explicitly drop support for the whole WPF (aka .NET 3.0) (aka WinFX) (aka WS-*) company-wide drive.?  Oh well, at least that’s one less choice to confuse me…but I have to bitch just this once: when can we abandon app-specific languages?  Any language or programming model that starts with the name of the vendor-specific technology makes me feel more than a little soiled.

Other resources that should prove useful

Some specific coding issues that I’ve noted for future reference

Interesting Tidbits

  • 🙂 Add-ins run out-of-proc in a hosted process called ehExtHost.exe, and communicate with the main Media Center process using .NET Remoting over “Named Pipes” (aka IPC within .NET).
  • 🙂 Media Center unloads the add-in when the Launch( ) method exit.
  • 🙂 The entry in More Programs is created by the running of RegisterMceApp.exe not by the running of the add-in itself.
  • 🙂 In an MCML application, you can only embed a couple of visual elements, NOT including ActiveX controls (i.e. no native Flash rendering).
  • 🙂 You can’t embed a WPF app / window inside of MCML.
  • 🙂 Charlie Owen has invited anyone to provide good/bad developer feedback on Media Center and the SDK, and left his contact info here.
  • 🙂 You can’t override ANYTHING on the Windows Media Center screens — if you want any new functions, they have to be added on completely new screens, which would mean re-building the whole ‘Videos’ library.

Conclusion

I’m not 100% convinced I’m going forward with this yet — I’m inclined to spec out the features, dig into the SDK/MCMLookalike/Codeplex-hosted sources to get a feel for just how much (a) volume and (b) complexity of code I’m facing, and then decide whether to keep pushing forward.  I’d sure appreciate feedback from anyone who has ever delved into any of these areas for encouragement or “there be dragons” warnings…

  • MCML programming
  • web services integration in managed code — Google or otherwise

Cheers all!