November 2006 - Posts

Introduction

Today (11/29/06) I received a mail from one of my blog readers:

Hi Bart,

in the following Block you mention an powershell commandlet for creating an SHA hash:

http://community.bartdesmet.net/blogs/bart/archive/2006/06/23/4106.aspx

Could you please publish or mail me some code snipes? My own code wont work proper.

THX for sharing so much knowledge about .NET.

Regards,

Thomas

Apparently I promised so time in the past to upload a cmdlet for file hashing but it never made it to my blog. So here it is today.

 

A file hasher cmdlet

Let's create a cmdlet for file hashing, called get-hash. It should take two parameters: one with the algorithm desired (SHA1, MD5, SHA256, SHA384, SHA512) and one with the file. The latter one can either be passed from the command line (e.g. dir *.cs | get-hash sha1 should work fine) or using some aliases specifying the name of the file as a string. Taking all these requirements together, we end up with the following:

1 using System; 2 using System.ComponentModel; 3 using System.IO; 4 using System.Management.Automation; 5 using System.Security.Cryptography; 6 using System.Text; 7 8 [Cmdlet("get", "hash")] 9 public class HashCmdlet : PSCmdlet 10 { 11 private string algorithm; 12 13 [Parameter(Position = 0, Mandatory = true)] 14 public string Algorithm 15 { 16 get { return algorithm; } 17 set { algorithm = value; } 18 } 19 20 private string file; 21 22 [Alias("File", "Name")] 23 [Parameter(Position = 1, Mandatory = true, ValueFromPipelineByPropertyName = true)] 24 public string FullName 25 { 26 get { return file; } 27 set { file = value; } 28 } 29 30 protected override void ProcessRecord() 31 { 32 HashAlgorithm algo = HashAlgorithm.Create(algorithm); 33 if (algo != null) 34 { 35 StringBuilder sb = new StringBuilder(); 36 using (FileStream fs = new FileStream(file, FileMode.Open)) 37 foreach(byte b in algo.ComputeHash(fs)) 38 sb.Append(b.ToString("x2")); 39 WriteObject(sb.ToString()); 40 } 41 else 42 { 43 string s = String.Format("Algorithm {0} not found.", algorithm); 44 ErrorRecord err = new ErrorRecord(new ArgumentException(s), s, ErrorCategory.InvalidArgument, null); 45 WriteError(err); 46 } 47 } 48 } 49 50 [RunInstaller(true)] 51 public class HashSnapin : PSSnapIn 52 { 53 public override string Name { get { return "FileHasher"; } } 54 public override string Vendor { get { return "Bart"; } } 55 public override string Description { get { return "Computes file hashes."; } } 56 }

A few remarks:

  • The second parameter, FullName, can be taken from the pipeline (ValueFromPipelineByPropertyName set to true). The reason for the chosen name "FullName" is the fact that a System.IO.FileInfo object has such a property name, and we want that property name to match our property name.
  • Using aliases, the second parameter can be supplied using File or Name too.
  • Line 39 write the string representation of the hash to the pipeline using WriteObject. Alternatively, one might also output the byte array retrieved from ComputeHash (line 37) but that would be less user-friendly in my opinion.
  • In case an unknown algorithm is specified, we construct an ErrorRecord object that's passed on to the shell using WriteError.
  • Stuff on lines 50 to 56 creates a simple snap-in that's used to make the cmdlet available.

 

Compilation and installation instructions

Download the code and execute the following steps on a VS2005 command line:

  • Make sure the PATH environment variable has csc.exe and installutil.exe on it (set PATH=%PATH%;%windir%\Microsoft.NET\Framework\v2.0.50727)
  • Save the file as hashcmdlet.cs
  • Copy System.Management.Automation from the GAC (%windir%\assembly\GAC_MSIL\System.Management.Automation\<version_pktoken>\System.Management.Automation.dll) to the current folder with the hashcmdlet.cs code file
  • Execute csc /t:library /r:System.Management.Automation.dll hashcmdlet.cs
  • Install the snap-in using installutil -i hashcmdlet.dll

 

Demo

Below you can see a sample of our get-hash cmdlet:

1 PS C:\temp> add-pssnapin filehasher 2 3 PS C:\temp> type extfi.ps1xml 4 <Types> 5 <Type> 6 <Name>System.IO.FileInfo</Name> 7 <Members> 8 <ScriptProperty> 9 <Name>MD5</Name> 10 <GetScriptBlock> 11 get-hash md5 $this 12 </GetScriptBlock> 13 </ScriptProperty> 14 <ScriptProperty> 15 <Name>SHA1</Name> 16 <GetScriptBlock> 17 get-hash sha1 $this 18 </GetScriptBlock> 19 </ScriptProperty> 20 </Members> 21 </Type> 22 </Types> 23 24 PS C:\temp> update-typedata extfi.ps1xml 25 26 PS C:\temp> dir *.cs | gm -type P* 27 28 TypeName: System.IO.FileInfo 29 30 Name MemberType Definition 31 ---- ---------- ---------- 32 PSChildName NoteProperty System.String PSChildName=bar.cs 33 PSDrive NoteProperty System.Management.Automation.PSDriveInfo PS... 34 PSIsContainer NoteProperty System.Boolean PSIsContainer=False 35 PSParentPath NoteProperty System.String PSParentPath=Microsoft.PowerS... 36 PSPath NoteProperty System.String PSPath=Microsoft.PowerShell.C... 37 PSProvider NoteProperty System.Management.Automation.ProviderInfo P... 38 Attributes Property System.IO.FileAttributes Attributes {get;set;} 39 CreationTime Property System.DateTime CreationTime {get;set;} 40 CreationTimeUtc Property System.DateTime CreationTimeUtc {get;set;} 41 Directory Property System.IO.DirectoryInfo Directory {get;} 42 DirectoryName Property System.String DirectoryName {get;} 43 Exists Property System.Boolean Exists {get;} 44 Extension Property System.String Extension {get;} 45 FullName Property System.String FullName {get;} 46 IsReadOnly Property System.Boolean IsReadOnly {get;set;} 47 LastAccessTime Property System.DateTime LastAccessTime {get;set;} 48 LastAccessTimeUtc Property System.DateTime LastAccessTimeUtc {get;set;} 49 LastWriteTime Property System.DateTime LastWriteTime {get;set;} 50 LastWriteTimeUtc Property System.DateTime LastWriteTimeUtc {get;set;} 51 Length Property System.Int64 Length {get;} 52 Name Property System.String Name {get;} 53 MD5 ScriptProperty System.Object MD5 {get=get-hash md5 $this;} 54 Mode ScriptProperty System.Object Mode {get=$catr = "";... 55 SHA1 ScriptProperty System.Object SHA1 {get=get-hash sha1 $this;} 56 57 PS C:\temp> dir *.cs | format-table Name,MD5,SHA1 58 59 Name MD5 SHA1 60 ---- --- ---- 61 bar.cs d541e9719077844ba1fa136... 8662e86f3302578a59da5e... 62 downloadfilecmdlet.cs 0c74a0c905f3b1cd6e22d52... ab3c4dcee4f9e3c48daded... 63 hashcmdlet.cs 41b01139d6168df3f3cec13... dd478c60f77b19b64fa0d7... 64 test.cs 477405d2be4a8f327d39a01... c632fe67a71baa0f333675... 65 66 PS C:\temp> dir *.cs | format-list Name,MD5,SHA1 67 68 Name : bar.cs 69 MD5 : d541e9719077844ba1fa13626f5122cb 70 SHA1 : 8662e86f3302578a59da5e9c936b69ab0d4ff9aa 71 72 Name : downloadfilecmdlet.cs 73 MD5 : 0c74a0c905f3b1cd6e22d52831b92b31 74 SHA1 : ab3c4dcee4f9e3c48daded97f01ee01e8c572a2a 75 76 Name : hashcmdlet.cs 77 MD5 : 41b01139d6168df3f3cec13b9663e633 78 SHA1 : dd478c60f77b19b64fa0d7c62944ec1b948419c9 79 80 Name : test.cs 81 MD5 : 477405d2be4a8f327d39a015db255fdf 82 SHA1 : c632fe67a71baa0f333675f5cdc16fc547772c33 83 84 PS C:\temp> get-hash MD5 bar.cs 85 d541e9719077844ba1fa13626f5122cb 86 87 PS C:\temp> get-hash SHA1 bar.cs 88 8662e86f3302578a59da5e9c936b69ab0d4ff9aa 89 90 PS C:\temp> get-hash SHA256 bar.cs 91 a1e6764cf77d02804e909427aff62ade6b9894924a69284f3d83fd0d2904548b 92 93 PS C:\temp> get-hash SHA384 bar.cs 94 875bde9e789f88e76aa9fe18f82adc8a8beb920cdf1d50692a0b9473ecc296a750e888844a184d7 95 6e610d434a3bec3a5 96 97 PS C:\temp> get-hash SHA512 bar.cs 98 f88b40bd4618dcb99e17af14c7f2368b00ea55a9b7d6d71e73d519e197ca3d6c3847fd46834fb1e 99 c4acb9c45729441ac76611de2f7f86b032b59e1a3b7384a3a 100 101 PS C:\temp> get-hash bla bar.cs 102 get-hash : Algorithm bla not found. 103 At line:1 char:9 104 + get-hash <<<< bla bar.cs

This sample is available in the download as well. Let's explain it in a bit more detail:

  • One line 1, the snap-in is loaded (which should be installed using installutil -i hashcmdlet.dll in the previous paragraph) using add-pssnapin.
  • Next, we leverage the power of the ETS (Extended Type System) to add two script properties MD5 and SHA1 to the FileInfo object. To do this, you should write an xml file which contains the line 4 to 22 and save it as a .ps1xml file. This file is loaded using update-typedata in line 24. This file is available in the download as well.
  • Now notice that FileInfo is extended with the MD5 and SHA1 ScriptProperty members using a get-member invocation on the output for get-childitem *.cs (assuming you have .cs files in your c:\temp folder). Lines 53 and 55 contain the newly added properties.
  • To use these script properties, observe the commands invoked in lines 57 and 66. These use format-table and format-list to visualize the additional properties.
  • Of course you can invoke get-hash directly too, as shown on lines 84, 87, 90, 93, 97 and 101. All supported algorithms are illustrated.

One drawback of our cmdlet is that it can't report progress when a hash operation takes a bit of time, especially for larger files. So, use it with care and rely on an explicit call to get-hash when you need to calculate a hash.

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

Two little-known features of ILASM (the IL assembler for .NET) are #define and .typedef which can reduce typing significantly, just as these do in a classic programming language. Often people do "round-tripping", i.e. they write an application in C#, ildasm it, make some slight modification, and ilasm it again. Now, in such a cycle you won't see all of the features of IL, such as #define and .typedef. So, if you're interested in IL, read on.

A few weeks ago I was creating a simple sample of "IL from scratch", meaning you take Notepad, write some IL and feed it to the ILASM tool. This was what I produced:

1 .assembly extern mscorlib {} 2 .assembly Demo {} 3 4 .namespace Bar.Foo 5 { 6 .class Demo 7 { 8 .field private static class Bar.Foo.Base printer 9 10 .method public static void Main() 11 { 12 .entrypoint 13 ldstr "Hello World" 14 newobj instance void Bar.Foo.Test::.ctor(string) 15 stsfld class Bar.Foo.Base Bar.Foo.Demo::printer 16 ldsfld class Bar.Foo.Base Bar.Foo.Demo::printer 17 call instance void Bar.Foo.Test::Print() 18 ret 19 } 20 } 21 22 .class abstract Base 23 { 24 .field family string name 25 26 .method public void .ctor(string name) 27 { 28 ldarg.0 //this 29 ldarg.1 //name 30 stfld string Bar.Foo.Base::name 31 ret 32 } 33 34 .method abstract virtual family void Print() 35 { 36 } 37 } 38 39 .class sealed Test extends Bar.Foo.Base 40 { 41 .method public void .ctor(string name) 42 { 43 ldarg.0 //this 44 ldarg.1 //name 45 call instance void Bar.Foo.Base::.ctor(string) 46 ret 47 } 48 49 .method public virtual void Print() 50 { 51 .override Bar.Foo.Base::Print 52 ldarg.0 //this 53 ldfld string Bar.Foo.Base::name 54 call [mscorlib]System.Console::WriteLine(string) 55 ret 56 } 57 } 58 }

As you can guess, this piece of code illustrates object-orientation in MSIL by creating some type with virtual methods, overriding, a family (~ protected) field, etc. However, there's quite a lot of typing to be done. For example, look at line 15 and 16 where the printer field is referenced including its type and its location. Basically, there is no such thing as namespaces in IL (line 4 is just syntactical sugar to say that all the classes below - line 6 and 39 - have to be prefixed with <namespace_name>. yielding Bar.Foo.Demo and Bar.Foo.Base. This same rule results in things like on line 54 where the assembly (mscorlib), the namespace (System.), the class (Console) and the method (::WriteLine) plus its arguments ((string)) have to written down to make a method call to the right overload. It would be much nicer to abbreviate these if you have to type it a lot.

#define will help us to reduce typing on lines 15 and 16. It's a precompiler directive that drives string replacement in what follows (just like the #ifdef...#else...#endif is a typical control directive). So, we can do something like this:

#define PRINTER "class Bar.Foo.Base Bar.Foo.Demo::printer"

.typedef is used for module-wide aliases. It allows to write stuff like:

.typedef [mscorlib]System.Console as Console .typedef method void Console::WriteLine(string) as WriteLine

So, we can just write call WriteLine instead of call [mscorlib]System.Console::WriteLine(string). The difference with a similar definition using a #define is the fact that aliases are more restrictive and can only be used for classes, methods, fields and attributes. Also, #define hasn't any meaning to the MSIL compiler itself since it's done in the pre-compilation phase. On the other hand, aliases are a part of the MSIL language and result in metadata, which means they can survive round-tripping. In other words, an alias defined in a piece of IL will remain in there even after a ILASM-ILDASM cycle of operations, whileas a #define will vanish during this process.

The end-result looks like this:

1 #define PRINTER "class Bar.Foo.Base Bar.Foo.Demo::printer" 2 3 .typedef [mscorlib]System.Console as Console 4 .typedef method void Console::WriteLine(string) as WriteLine 5 6 .assembly extern mscorlib {} 7 .assembly Demo {} 8 9 .namespace Bar.Foo 10 { 11 .class Demo 12 { 13 .field private static class Bar.Foo.Base printer 14 15 .method public static void Main() 16 { 17 .entrypoint 18 ldstr "Hello World" 19 newobj instance void Bar.Foo.Test::.ctor(string) 20 stsfld PRINTER 21 ldsfld PRINTER 22 call instance void Bar.Foo.Test::Print() 23 ret 24 } 25 } 26 27 .class abstract Base 28 { 29 .field family string name 30 31 .method public void .ctor(string name) 32 { 33 ldarg.0 //this 34 ldarg.1 //name 35 stfld string Bar.Foo.Base::name 36 ret 37 } 38 39 .method abstract virtual family void Print() 40 { 41 } 42 } 43 44 .class sealed Test extends Bar.Foo.Base 45 { 46 .method public void .ctor(string name) 47 { 48 ldarg.0 //this 49 ldarg.1 //name 50 call instance void Bar.Foo.Base::.ctor(string) 51 ret 52 } 53 54 .method public virtual void Print() 55 { 56 .override Bar.Foo.Base::Print 57 ldarg.0 //this 58 ldfld string Bar.Foo.Base::name 59 call WriteLine 60 ret 61 } 62 } 63 }

In name of all IL freaks, enjoy MSIL 2.0!

kick it on DotNetKicks.com

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

Only a couple of days ago I posted about creating a file downloader cmdlet in Windows PowerShell which contained the following little sentence:

One could make this method more complex in order to provide a seconds remaining estimate based on the download speed observed.

One of my readers (dotnetjunkie) was so kind to leave a piece of feedback:

I think you should add transfer speed and estimated remaining time, that would make it even more useful and cooler ! ;)

Well, I couldn't agree more. So I revisited my piece of code I wrote some weeks ago (12th of November to be precise) and added download speed tracking and a seconds remaining indicator.

There are undoubtly different ways to implement such an estimate. My approach is to measure the number of bytes transferred during intervals of approximately 5 seconds (kind of "instant download speed") and to derive the estimated time remaining from this. I'll discuss the code changes in a few steps.

Step 1 - Add a few private members

We need 4 additional members to produce statistics:

  • The first one is a Stopwatch to measure elapsed time. We'll poll this counter every time the DownloadProgressChanged event is fired. In case it's above 5 seconds, we update the statistics and restart the counter.
  • Secondly, we'll cache an indicator that keeps the current transfer speed as a string of the format n [bytes|KB|MB|GB|...]/sec. This value will be updated every 5 seconds.
  • Next, the seconds remaining are kept. It's updated every 5 seconds and during these intervals it just counts down every one second.
  • Finally, a transferred bytes indicator is used for calculation of the bytes transferred the last 5 seconds.

Here's the piece of code:

/// <summary> /// Stopwatch used to measure download speed. /// </summary> private Stopwatch sw = new Stopwatch(); /// <summary> /// Bytes per second indicator (bytes/sec, KB/sec, MB/sec, ...). /// </summary> private string bps = null; /// <summary> /// Seconds remaining indicator. /// </summary> private int secondsRemaining = -1; /// <summary> /// Number of bytes already transferred. /// </summary> private long transferred = 0;

Step 2 - Let the count begin

In the ProcessRecord method we start our Stopwatch; just that:

// // Check validity for download. Will throw an exception in case of transport protocol errors. // using (clnt.OpenRead(_url)) { } // // Start download speed stopwatch. // sw.Start(); // // Download the file asynchronously. Reporting will happen through events on background threads. // clnt.DownloadFileAsync(_url, _file);

Step 3 - Calculate stats and report progress

Time for the real stuff. On to the DownloadProgressChanged event handler. When we observe that the Stopwatch has an elapsed time of 5 or more seconds, we'll stop it, update stats and restart it. The code is shown below:

1 /// <summary> 2 /// Reports download progress. 3 /// </summary> 4 private void webClient_DownloadProgressChanged(object sender, DownloadProgressChangedEventArgs e) 5 { 6 // 7 // Update statistics every 5 seconds (approx). 8 // 9 if (sw.Elapsed >= TimeSpan.FromSeconds(5)) 10 { 11 sw.Stop(); 12 13 // 14 // Calculcate transfer speed. 15 // 16 long bytes = e.BytesReceived - transferred; 17 double bps = bytes * 1000.0 / sw.Elapsed.TotalMilliseconds; 18 this.bps = BpsToString(bps); 19 20 // 21 // Estimated seconds remaining based on the current transfer speed. 22 // 23 secondsRemaining = (int)((e.TotalBytesToReceive - e.BytesReceived) / bps); 24 25 // 26 // Restart stopwatch for next 5 seconds. 27 // 28 transferred = e.BytesReceived; 29 sw.Reset(); 30 sw.Start(); 31 } 32 33 // 34 // Construct a ProgressRecord with download state information but no completion time estimate (SecondsRemaining < 0). 35 // 36 ProgressRecord pr = new ProgressRecord(0, String.Format("Downloading {0}", _url.ToString(), _file), String.Format("{0} of {1} bytes transferred{2}.", e.BytesReceived, e.TotalBytesToReceive, this.bps != null ? String.Format(" (@ {0})", this.bps) : "")); 37 pr.CurrentOperation = String.Format("Destination file: {0}", _file); 38 pr.SecondsRemaining = secondsRemaining - (int)sw.Elapsed.Seconds; 39 pr.PercentComplete = e.ProgressPercentage; 40 41 // 42 // Report availability of a ProgressRecord item. Will cause the while-loop's body in ProgressRecord to execute. 43 // 44 lock (pr_sync) 45 { 46 this.pr = pr; 47 prog.Set(); 48 } 49 }

So, what's going on here. Basically we want to provide a seconds remaining estimate on line 38 and a download speed estimate on line 36. This should be pretty self-explanatory. The real work happens in lines 11 to 30 where the number of bytes transferred in the last 5 seconds are obtained and divided by the expired milliseconds during the last 5 seconds (which should be around 5000 obviously). The rest is maths, except for the BpsToString call as shown below.

Step 4 - A download speed indicator

BpsToString is the method to convert the bytes per second rate to a friendly string representation:

/// <summary> /// Constructs a download speed indicator string. /// </summary> /// <param name="bps">Bytes per second transfer rate.</param> /// <returns>String represenation of the transfer rate in bytes/sec, KB/sec, MB/sec, etc.</returns> private string BpsToString(double bps) { string[] m = new string[] { "bytes", "KB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB" }; //dreaming of YB/sec int i = 0; while (bps >= 0.9 * 1024) { bps /= 1024; i++; } return String.Format("{0:0.00} {1}/sec", bps, m[i]); }

I think the code fragment above is pretty optimistic for what transfer speeds is concerned, but with the expected life time of PowerShell in mind this should be no luxury :-).

Step 5 - The result & code download

This is the result (needless to say the figures are indicative only, it are estimates after all):

And here's the code download link.

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

A few days ago, I posted this little quiz on batch scripting mysteries. The batch script given was:

@echo off for %%f in (a,b,c) do ( echo 1 %%f 2 set x=x1 %%f x2 echo %x% )

Here's the output:

C:\temp>test
1 a 2
ECHO is off.
1 b 2
ECHO is off.
1 c 2
ECHO is off.

C:\temp>test
1 a 2
x1 c x2
1 b 2
x1 c x2
1 c 2
x1 c x2

Not what you might have expected :-). I did some research but HELP SET didn't give the complete answer, so a workaround seems a good solution:

@echo off for %%f in (a,b,c) do ( echo 1 %%f 2 call :print %%f ) goto eof :print set x=x1 %1 x2 echo %x% :eof

And of course Windows PowerShell helps us out too:

PS C:\Users\Bart> ('a','b','c') | foreach { echo "1 $_ 2"; $x = "x1 $_ x2"; echo $x } 1 a 2 x1 a x2 1 b 2 x1 b x2 1 c 2 x1 c x2

Yet another reason to choose Windows PowerShell? (Notice the variable expansion in strings too.)

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

Introduction

In my previous post on file downloads in PowerShell I mentioned how to download a file in Windows PowerShell using System.Net.WebClient. No big deal if you know the Base Class Libraries. However, there was a drawback: the lacking download status reporting. In order to fix this, I've created a simple sample cmdlet that uses reporting (WriteProgress) while performing a download. Enter download-file.

The code

The basic idea is pretty simple:

  • Inside a cmdlet we'll call some WebClient method to perform a file download from a specified URL (through some parameter) and to a local file location (which can be an optional parameter, since we can derive the file name from the source URL as well)
  • Instead of blocking on some DownloadFile call, we want to do things asynchronously (i.e. DownloadFileAsync) in order to be able to report progress during the download.
  • The events DownloadProgressChanged and DownloadFileCompleted of the System.Net.WebClient will help us to accomplish the progress reporting.

We'll call our cmdlet download-file, so the declaration will be like this:

[Cmdlet("download", "file")] public class DownloadFileCmdlet : PSCmdlet { }

Parameterization of a cmdlet is piece of cake as explained in a bunch of previous posts already, and will look like this:

/// <summary> /// Url of the file that has to be downloaded. /// </summary> [Parameter(Mandatory = true, Position = 0)] public Uri Url { get { return _url; } set { _url = value; } } private string _file; /// <summary> /// Target file name (optional). /// </summary> [Parameter(Position = 1)] public string To { get { return _file; } set { _file = value; } }

Next, we'll have to define the ProcessRecord overload of the cmdlet to do the processing. This is where things get tricky because of the following reasons:

  • Starting an asynchronous download using DownloadFileAsync doesn't report HTTP error codes directly. Since we want to catch these possible errors, we'll have to do some trick.
  • Events like DownloadProcessChanged and DownloadFileCompleted are raised on a background thread. From such a thread it's not valid to call the PSCmdlet's WriteProgress (or any other Write*) method.
  • We have to wait for the background download to complete before ProcessRecord is exited. In other words, DownloadFileCompleted has to run before we can exit ProcessRecord.

The solution to all of the problems above is some nice piece of thread synchronization. Basically the "main thread" (the one where ProcessRecord was called on) has to create the WebClient, hook in event handlers and start the asynchronous download job. Once that's done, it has to wait for any of two events to occur: either a ProgressRecord instance is available or DownloadFileCompleted has executed. In the first case, we can perform a WriteProgress to report progress on the right thread. In the second case, we can exit the ProcessRecord method because of download completion.

Here's the complete code:

1 using System; 2 using System.ComponentModel; 3 using System.IO; 4 using System.Management.Automation; 5 using System.Net; 6 using System.Threading; 7 8 [Cmdlet("download", "file")] 9 public class DownloadFileCmdlet : PSCmdlet 10 { 11 /// <summary> 12 /// Wait handle to report download completion. 13 /// </summary> 14 private ManualResetEvent exit = new ManualResetEvent(false); 15 16 /// <summary> 17 /// Wait handle to report availability of a ProgressRecord item in pr. 18 /// </summary> 19 private AutoResetEvent prog = new AutoResetEvent(false); 20 21 /// <summary> 22 /// Array of the wait handles above (set in ProcessRecord) to perform WaitAny. 23 /// </summary> 24 private WaitHandle[] evts; 25 26 /// <summary> 27 /// ProgressRecord indicating the current download status. 28 /// </summary> 29 private ProgressRecord pr; 30 31 /// <summary> 32 /// Synchronization object for pr. 33 /// </summary> 34 private object pr_sync = new object(); 35 36 private Uri _url; 37 38 /// <summary> 39 /// Url of the file that has to be downloaded. 40 /// </summary> 41 [Parameter(Mandatory = true, Position = 0)] 42 public Uri Url 43 { 44 get { return _url; } 45 set { _url = value; } 46 } 47 48 private string _file; 49 50 /// <summary> 51 /// Target file name (optional). 52 /// </summary> 53 [Parameter(Position = 1)] 54 public string To 55 { 56 get { return _file; } 57 set { _file = value; } 58 } 59 60 /// <summary> 61 /// Entry-point for the cmdlet processing. 62 /// </summary> 63 protected override void ProcessRecord() 64 { 65 // 66 // Construct wait handles array for WaitHandle.WaitAny calls. 67 // 68 evts = new WaitHandle[] { exit, prog }; 69 70 // 71 // If no target file name was specified, derive it from the url's file name portion. 72 // 73 if (_file == null) 74 { 75 string[] fs = _url.LocalPath.Split('/'); 76 if (fs.Length > 0) 77 _file = fs[fs.Length - 1]; 78 } 79 80 // 81 // Construct web client object and hook in event handlers to report progress and completion. 82 // 83 WebClient clnt = new WebClient(); 84 clnt.DownloadProgressChanged += new DownloadProgressChangedEventHandler(webClient_DownloadProgressChanged); 85 clnt.DownloadFileCompleted += new AsyncCompletedEventHandler(webClient_DownloadFileCompleted); 86 87 try 88 { 89 // 90 // Check validity for download. Will throw an exception in case of transport protocol errors. 91 // 92 using (clnt.OpenRead(_url)) { } 93 94 // 95 // Download the file asynchronously. Reporting will happen through events on background threads. 96 // 97 clnt.DownloadFileAsync(_url, _file); 98 99 // 100 // Wait for any of the events (exit, prog) to occur. 101 // In case of index 0 (= exit), stop processing. 102 // In case of index 1 (= prog), report progress. 103 // 104 while (WaitHandle.WaitAny(evts) != 0) //0 is exit event 105 { 106 lock (pr_sync) 107 { 108 WriteProgress(pr); 109 } 110 } 111 112 // 113 // Write file info object for the target file. Can be used for further processing on the pipeline. 114 // 115 WriteObject(new FileInfo(_file)); 116 } 117 catch (WebException ex) 118 { 119 // 120 // Report an error. Could be more specific for what the ErrorCategory is concerned, by mapping HTTP error codes. 121 // 122 WriteError(new ErrorRecord(ex, ex.Status.ToString(), ErrorCategory.NotSpecified, clnt)); 123 } 124 } 125 126 /// <summary> 127 /// Reports download progress. 128 /// </summary> 129 private void webClient_DownloadProgressChanged(object sender, DownloadProgressChangedEventArgs e) 130 { 131 // 132 // Construct a ProgressRecord with download state information but no completion time estimate (SecondsRemaining < 0). 133 // 134 ProgressRecord pr = new ProgressRecord(0, String.Format("Downloading {0}", _url.ToString(), _file), String.Format("{0} of {1} bytes transferred.", e.BytesReceived, e.TotalBytesToReceive)); 135 pr.CurrentOperation = String.Format("Destination file: {0}", _file); 136 pr.SecondsRemaining = -1; 137 pr.PercentComplete = e.ProgressPercentage; 138 139 // 140 // Report availability of a ProgressRecord item. Will cause the while-loop's body in ProgressRecord to execute. 141 // 142 lock (pr_sync) 143 { 144 this.pr = pr; 145 prog.Set(); 146 } 147 } 148 149 /// <summary> 150 /// Reports download completion. 151 /// </summary> 152 private void webClient_DownloadFileCompleted(object sender, System.ComponentModel.AsyncCompletedEventArgs e) 153 { 154 // 155 // Signal the exit state. Will cause the while-loop in ProcessRecord to terminate. 156 // 157 exit.Set(); 158 } 159 } 160 161 [RunInstaller(true)] 162 public class DownloadFileSnapIn : PSSnapIn 163 { 164 public override string Name { get { return "DownloadFile"; } } 165 public override string Vendor { get { return "Bart De Smet"; } } 166 public override string Description { get { return "Allows file download."; } } 167 }

Starting at the bottom of the cmdlet definition we can see the two event handlers, webClient_DownloadProgressChanged and webClient_DownloadFileCompleted. The first one creates a ProgressRecord (lines 134-137) which is the way for a cmdlet to communicate status to the PowerShell host application. The parameters and properties are self-explanatory. One could make this method more complex in order to provide a seconds remaining estimate based on the download speed observed. In this basic sample we're happy with some status messages and a percentage. In order to report progress, some thread synchronization stuff is needed. Remember the WriteProgress method can only be called on the ProcessRecord's thread. So, we copy (line 144) the constructed ProgressRecord to the pr member of the class (line 29) which is synchronized by pr_sync (line 34, used in line 142). Finally the availability of the record is signaled using the wait handle prog in line 145. Notice it's an AutoResetEvent (line 19), which means that it gets reset automatically (to false) once the consuming thread (ProcessRecord) has sucked it (in WaitAny in our case, line 104). The webClient_DownloadFileCompleted event handler is straightforward an just signals the exit "download completion" state on line 157 that will be caught on line 104's WaitAny.

Real work happens in ProcessRecord of course. First (line 68) an array of wait handles gets constructed to be used in the WaitAny call (line 104). WaitAny means to wait till either one of these handles has been set. For example: if a progress record is available, prog will be set (by webClient_DownloadProgressChanges on line 145) and WaitAny will return 1 because prog is on index 1 in the evts handles array. In a similar way, WaitAny will return 0 if the 0'th element of evts, i.e. exit, has been set (by webClient_DownloadFileCompleted on line 157). Next, the case of no specified target file is taken care of in lines 73 to 78, taking the source file name as the target name. Now (lines 83-85) the WebClient instance is created and the event handlers are hooked in. Finally, the download progress can be started. In order to cause an exception - for example in case of a 404 error code - before download begins, we call OpenRead on line 92. The asynchronous download job is started on line 97.

The summum of our code is in lines 104-110, where ProgressRecord instances are consumed every time the prog wait handle is set. These are reported to the PowerShell host by means of a WriteProgress call (line 108), taking care of the required locking. Finally, in line 115, a FileInfo object for the target file is written on the pipeline which might be useful for further processing. WebExceptions are caught on line 117 and reported via WriteError on line 122.

The snap-in for the cmdlet goes without further explanation, see lines 161 to 167.

 

Compilation

Compiling this goes as follows:

  • Make sure the PATH environment variable has csc.exe and installutil.exe on it (set PATH=%PATH%;%windir%\Microsoft.NET\Framework\v2.0.50727)
  • Save the file as downloadfilecmdlet.cs
  • Copy System.Management.Automation from the GAC (%windir%\assembly\GAC_MSIL\System.Management.Automation\<version_pktoken>\System.Management.Automation.dll) to the current folder with the downloadfilecmdlet.cs code file
  • Execute csc /t:library /r:System.Management.Automation.dll downloadfilecmdlet.cs
  • Install the snap-in using installutil -i downloadfilecmdlet.dll

 

Demo

See the pictures below for the cmdlet in action (H):

Download the code over here.

Have fun!

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

From the code fragment below non-Dutch speaking readers can already predict what I'm building for the moment. For the Dutch folks, there's something funny in the translated C# compiler errors:

downloadfilecmdlet.cs(38,10): error CS1502: De beste treffer voor de overbelaste methode voor System.Management.Automation.Cmdlet.WriteError(System.Management.Automation.ErrorRecord) heeft enkele ongeldige argumenten

(I'm currently working on a machine which has the Dutch .NET Framework 2.0 installed with now developer tools, and so I'm doing some command-line compiles.)

This reminds me of the "lower case" translation into "onderkast" (see a pretty old post on that one).

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

Time for a little PowerShell tip for my IT Pro friends. I know it's just BCL (Base Class Library) stuff, but nevertheless you might find it useful: what about downloading a file from an HTTP server during script execution? Here it is:

PS C:\temp> $clnt = new-object System.Net.WebClient
PS C:\temp> $clnt | gm d*


   TypeName: System.Net.WebClient

Name                MemberType Definition
----                ---------- ----------
Dispose             Method     System.Void Dispose()
DownloadData        Method     System.Byte[] DownloadData(String address), S...
DownloadDataAsync   Method     System.Void DownloadDataAsync(Uri address), S...
DownloadFile        Method     System.Void DownloadFile(String address, Stri...
DownloadFileAsync   Method     System.Void DownloadFileAsync(Uri address, St...
DownloadString      Method     System.String DownloadString(String address),...
DownloadStringAsync Method     System.Void DownloadStringAsync(Uri address),...


PS C:\temp> $url = "http://www.bartdesmet.net/download/ps.txt"
PS C:\temp> $file = "c:\temp\ps.txt"
PS C:\temp> $clnt.DownloadFile($url,$file)
PS C:\temp> type $file
Welcome to Windows PowerShell 1.0! 

Drawback to this approach: no download reporting (as with write-progress) while downloading a large file. Solution: coming up later. Enjoy!

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

Time to pre-empt some other posts for a great Channel 9 video with Anders Hejlsberg and Chris McConnell on LINQ, Desktop Search, WinFS, Functional and Intentional Programming. Interesting view and stuff to think about.

Watch it here.

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

Introduction

In the previous posts on DynCalc, we explored how to write a simple parser for simple calculations, transforming it from infix to postfix and finally to a tree that can be interpreted or compiled for execution of the calculation.

In a short summary, we did the following:

  • Infix to postfix: The input expression ((1+2)+3)*2-8/4 is translated into 1 2 + 3 + 2 * 8 4 / -.
  • Postfix to expression tree: 1 2 + 3 + 2 * 8 4 / - is translated into a tree representation like this:

    +Sub
     +Mul
      +Add
       +Add
        1
        2
       3
      2
     +Div
      8
      4

  • Expression tree to IL: The real dynamic compilation turns the tree above into the following piece of IL, ready for execution:

    ldc.i4.s 1
    ldc.i4.s 2
    add
    ldc.i4.s 3
    add
    ldc.i4.s 2
    mul
    ldc.i4.s 8
    ldc.i4.s 4
    div
    add

As of C# 3.0, there's library support for expression trees and compilation of these. In this post, we'll transform our DynCalc sample into an equivalent application using C# 3.0 Expression Trees.

 

Building the Expression Tree

As illustrated in Part 2: Building an expression tree we take a queue of MathOpOrVal tokens (representing either a mathematical operation or an integer value) and turn it into an expression tree as follows:

static TreeNode ToTree(Queue<MathOpOrVal> q) { Stack<TreeNode> stack = new Stack<TreeNode>(); foreach (MathOpOrVal mv in q) { if (mv.Value != null) stack.Push(new TreeNode(mv.Value.Value)); else { TreeNode right = stack.Pop(); TreeNode left = stack.Pop(); stack.Push(new TreeNode(mv.Op.Value, left, right)); } } return stack.Pop(); }

In here, the TreeNode was an internal class to represent a tree node consisting of an operation together with a left and right node. All of this can be replaced by the C# 3.0 Expression Trees, as follows:

1 static Expression<Func<int>> ToTree2(Queue<MathOpOrVal> q) 2 { 3 Stack<Expression> stack = new Stack<Expression>(); 4 5 foreach (MathOpOrVal mv in q) 6 { 7 if (mv.Value != null) 8 stack.Push(Expression.Constant(mv.Value.Value)); 9 else 10 { 11 Expression right = stack.Pop(); 12 Expression left = stack.Pop(); 13 switch (mv.Op.Value) 14 { 15 case MathOp.Add: 16 stack.Push(Expression.Add(left, right)); 17 break; 18 case MathOp.Sub: 19 stack.Push(Expression.Subtract(left, right)); 20 break; 21 case MathOp.Mul: 22 stack.Push(Expression.Multiply(left, right)); 23 break; 24 case MathOp.Div: 25 stack.Push(Expression.Divide(left, right)); 26 break; 27 case MathOp.Mod: 28 stack.Push(Expression.Modulo(left, right)); 29 break; 30 } 31 } 32 } 33 34 return Expression<Func<int>>.Lambda<Func<int>>( 35 stack.Pop(), 36 new ParameterExpression[0] 37 ); 38 }

Let's explain this code step-by-step:

  • The signature (line 1) indicates we consume the same queue as we did before, but this time we return an Expression (namespace System.BLOCKED EXPRESSION supplied by the Orcas C# 3.0 Framework. The Expression<Func<int>> indicates that the expression represents a function with return type int. Func<int> is a generic type "instance" of Func<R> in which R stands for "return type". Basically Func<R> is a delegate with signature public delegate R Func<R>();. In case the expression tree would consume a parameter, one could use Expression<Func<int,int>> or Expression<Func<int,int,int>> (2 int parameters).
  • Instead of building a stack of custom tree nodes, we now build a Stack<Expression> as shown in line 3.
  • Next, we iterate over the queue in lines 5 to 32.
    • In case the value of the MathOpOrVal object in the queue is set, we have to add a constant value to the stack, which is constructed via the factory method Expression.Constant in line 8.
    • Otherwise an operation will be set. In this case, we have to pop both arguments to the binary operation from the stack (lines 11, 12) and do case analysis based on the binary operation desired (lines 13 to 30). To represent a binary operation in the tree, one uses the Expression.<operation>(Expression left, Expression right) factory method.
  • Finally (line 34), the resulting expression is popped from the stack and wrapped in a lambda expression that can be returned. Notice the use of ParameterExpression[0] to indicate no parameters are required.

The cool thing of the built expression is that it doesn't require manual compilation as we did in Part 3: Compilation to IL. Instead we can just call Compile() on the expression. To illustrate this, the Main method is changed as follows:

1 static void Main(string[] args) 2 { 3 Console.WriteLine("Dynamic calculator"); 4 Console.WriteLine("------------------"); 5 Console.WriteLine(); 6 7 Console.Write("Expression: "); 8 string expr = Console.ReadLine(); 9 10 Queue<MathOpOrVal> q = Expr.InfixToPostfix(expr); 11 12 Console.WriteLine(); 13 Console.Write("Postfix representation: "); 14 Print(q); 15 16 Console.WriteLine(); 17 Console.WriteLine("Tree representation:"); 18 //TreeNode tree = ToTree(q); 19 //Print(tree); 20 Expression<Func<int>> f = ToTree2(q); 21 StringBuilder sb = new StringBuilder(); 22 f.BuildString(sb); 23 Console.WriteLine(sb.ToString()); 24 25 Console.WriteLine(); 26 Console.Write("Dynamic calculation: "); 27 //Console.WriteLine("Result = {0}", Execute(tree)); 28 Console.WriteLine("Result = {0}", f.Compile().Invoke()); 29 }

As you can see (line 28), execution is piece of cake thanks to the System.Expressions library. Also, printing the tree is relatively straightforward using a StringBuilder (lines 20 to 23). A sample execution of this application is shown below:

Dynamic calculator ------------------ Expression: ((1+2)+3)*2-8/4 Postfix representation: 1 2 Add 3 Add 2 Mul 8 4 Div Sub Tree representation: () => Subtract(Multiply(Add(Add(1, 2), 3), 2), Divide(8, 4)) Dynamic calculation: Result = 10

Notice the equivalence of our tree representation, i.e.

+Sub
 +Mul
  +Add
   +Add
    1
    2
   3
  2
 +Div
  8
  4

and the C# 3.0 Expression Tree lambda expression, i.e.

() => Subtract(Multiply(Add(Add(1, 2), 3), 2), Divide(8, 4))

Stay tuned for even more Expression Tree fun in the future.

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

Somewhere in October this year, a series of three posts appeared online on this blog, covering "DynCalc", focusing on dynamic compilation:

Strictly spoken, this post is part of this series. The first three parts of this posting series were meant to provide the introduction to expression trees in C# 3.0. In this post, I'll introduce C# 3.0 Expression Trees very briefly. A future post will connect the DynCalc idea to C# 3.0 Expression Trees further on.

C# 3.0 focuses on one big innovation: LINQ (Language INtegrated Query), which has been discussed previously on this blog as well. The basic idea is to close the gap between all sorts of data representations, including objects, relational data and hierarchical data (XML). In order to provide developers an integrated query syntax in the language itself (which leads to type safety, catching errors at compile time, etc instead of the opaque SQL statements inside some string), the language has been extended with (context-sensitive, in order to maintain backward compatibility) keywords like from, where, orderby, etc. A typical example is shown below:

var res = from p in System.Diagnostics.Process.GetProcesses() where p.HandleCount > 100 select new { Name = p.ProcessName, p.WorkingSet }; foreach (var i in res) { Console.WriteLine("{0,-30}{1}", p.Name, p.WorkingSet); }

The first line contains quite some C# 3.0 features:

  • Local type inference with the var keyword, which is in this case a requirement since we construct an anonymous type (see third bullet).
  • LINQ syntax to formulate a query selecting objects (LINQ-to-Objects).
  • Anymous type creation using the new { ... } syntax.

For this discussion, it's important to see that the first line - the LINQ query - is compiled to IL code directly. Basically, the compiler rewrites the line as:

var res = System.Diagnostics.Process.GetProcesses().Where(p => p.HandleCount > 100).Select(p => new { Name = p.ProcessName, p.WorkingSet});

In here, extension methods are used on IEnumerable<T>, provided by the System.Query namespace's Sequence class. In reality this code is equivalent to:

var res1 = System.Query.Sequence.Where(System.Diagnostics.Process.GetProcesses(), p => p.HandleCount > 100); var res = System.Query.Sequence.Select(res1, p => new { Name = p.ProcessName, p.WorkingSet });

Which, on its turn is translated to get rid of the lamba expressions (which become anonymous methods) and the anymous type syntax (which becomes a constructor call to some auto-generated private class, together with two property setter calls). However, if one wants to run a query against a relational source, we don't want such an extensive compilation to IL. Instead we want some kind of tree that represents the query but can be translated (at runtime) into some domain-specific querying language like SQL or XQuery or LDAP or WQL or ..., you get the idea (strictly spoken I should mention IQueryable<T> here, but I'll defer that discussion to a future post). This is where expression trees enter the picture.

In the code fragment below, you can see a first example on how expression trees are created for lambda expressions. Instead of just doing something like Func<int> f = () => 2 * 3 + 1; (which results in a compiled method which can be called like int res = f();) one can wrap the declaration in an Expression<T> object. This makes the C# compiler generate an expression tree instead of performing direct compilation. Next, you could parse the tree yourself manually, or request compilation at runtime by calling Compile(). Below, you can see how the expression tree for 2 * 3 + 1 can be created directly or manually (which we'll do later when combining this with our DynCalc "front-end").

using System; using System.Collections.Generic; using System.Text; using System.Query; using System.Expressions; namespace ModernDynCalc { class Program { static void Main(string[] args) { Expression<Func<int>> f = () => 2 * 3 + 1; int res = f.Compile().Invoke(); Console.WriteLine(res); Expression<Func<int>> f2 = Expression<Func<int>>.Lambda<Func<int>>( Expression.Add( Expression.Multiply( Expression.Constant(2), Expression.Constant(3) ), Expression.Constant(1) ), new ParameterExpression[0] ); int res2 = f2.Compile().Invoke(); Console.WriteLine(res2); } } }

In a next post, we'll combine this with our DynCalc effort, and yet later, a more extensive sample will be shown on how to use a "LINQ expression tree" to translate a query into some target query statement. Enjoy!

kick it on DotNetKicks.com

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

More Posts Next page »