markw.dev - the blog2022-02-02https://markw.devMark Wilkinsonme@markw.dev
{ what i learned giving my first presentation2014-05-20first-presentationSQL Saturday 292, Detroit - There is a real sense of community in the SQL world, and it really shows at events like this.<p>Recently I had the privilege of speaking at PASS SQL Saturday #292 in Detroit; I presented a session on the importance of monitoring: <a href="http://sqlsaturday.com/viewsession.aspx?sat=292&sessionid=20890">What Just Happened?</a>. This was my first time presenting to a room of people I had never met before, it was also the first SQL Saturday I had ever attended, and it was a great experience. There is a real sense of community in the SQL world, and it really shows at events like this.</p>
<p>Grant Fritchey mentioned this in both of his sessions, but it really needs to be repeated: All of the attendees/volunteers (and most of the speakers) at SQL Saturday are not being paid, and are sacrificing their own free time to teach and learn about SQL Server on a Saturday. As Grant put it, this makes those attendees some of the most valuable people in the market. They are people that are willing to give up some of their own time to better themselves and learn something new.</p>
<h1>What I Learned/What Worked for Me</h1>
<p>My session went better than I could have hoped, I got some great attendee feedback (both positive and constructive) and look forward to speaking at future SQL Saturdays and user group meetings. While preparing for and delivering the presentation, and while listening to other presenters, I learned a lot about what makes for a good session, and what I shouldn't have wasted my time on.</p>
<h2>Be Gracious</h2>
<p>Thank EVERYBODY. Without everybody at the event, you wouldn't be there. Every single person at these events is important. Without the hard work of the volunteers and your fellow speakers, there wouldn't be an event. Without the time sacrifice of your audience, you wouldn't have anyone to present to. Don't just remember this in the back of your mind, actually go and thank these people. Shake their hands and thank them.</p>
<h2>Ask Other Speakers What to Expect</h2>
<p>Ask around and see who the typical audience is in a "beginner" session. I thought I knew, but after starting my presentation I found out I was wrong. I found out that my definition of "beginner" might be a little closer to intermediate. When I started asking questions I was getting fewer raised hands and head nods than I thought I might, this actually made me more excited to present. It was cool to know that some of the attendees were being exposed to brand new concepts in my session. This was unexpected but in my mind it was a bonus.</p>
<h2>Do Your Research</h2>
<p>As I said above, you might have the opportunity to expose people to ideas that are brand new to them; make sure you get it right. While it is perfectly fine to answer a question with "I am not sure, I will have to get back to you on that." it isn't perfectly fine to guess and possibly give someone bad information.</p>
<h2>Check in with Your Audience</h2>
<p>Get a pulse on your audience from time to time, reel them back into the presentation. Before I would start covering a new topic I would poll the audience to see how many of them were familiar with it. Not only does this bring the attention of the audience back to you, it can also help you determine how you are going to cover the next topic. For example, many of the attendees in my session had never heard of wait statistics, so it wouldn't make much sense for me to launch right into a discussion of the sys.dm_os_wait_stats DMV without first explaining a bit about what it means when a query is "waiting".</p>
<h2>Break Things Up</h2>
<p>I am always a fan of well-placed humor, and it can do wonders to revitalize the audience. If humor isn't your thing, try to get some other type of interaction out of the audience. Ask questions about situations they have been in that are similar to those you are presenting, ask them for guesses "What do you think lock wait types tell us?", or as I mentioned above just ask them if they are familiar with the concept you are covering.</p>
<h2>Be Prepared</h2>
<p>Get to the venue early. Make sure you have enough time to get everything open and ready. Nobody in the audience wants to watch you fiddle around waiting for SSRS to spool up, or watch you frantically searching your hard drive for that missing demo script. On the same note, make sure you have a backup if your demo doesn't work. If your demo bombs, at least make sure to have some screen shots to show what the audience would have seen.</p>
<h2>Don't Over Prepare</h2>
<p>Preparation is great, do as much research as you need. Just don't try to write down every last word you plan on saying. If you spend time trying to "remember your line" you will likely just end up with a lot of "uh..." and "um..." moments in your presentation. If you instead just focus on KNOWING the topic, you can focus on the facts and worry less about exactley how to say it. Presenting is just talking, but to a larger group of people than you might be used to.</p>
<h2>Be Gracious</h2>
<p>No, this isn't a typo. This is a point that needs to be repeated. THANK EVERYBODY. Thank the attendees when they come in, thank them when they leave, thank them when they ask questions, or when they correct something you just said, thank the volunteers, thank the vendors, thank the local user group president that put it all together... THANK EVERYBODY.</p>
<h1>Now Go Present</h1>
<p>I highly recommend signing up to present a session at your next SQL Saturday, or User Group meeting. Don't assume your ideas would be "dumb" or "not technical enough". Attendees come in at every level of expertise possible, from "this is my first day on the job" to "I have been a production DBA for the past 10 years". Just find a topic you are passionate about and submit it. If you are passionate about something, and you feel like you could contribute to someone else’s understanding of it, submit a session and see what happens.</p>visualizing statistics in ssms2014-09-11visualizing-statistics-in-ssms-2Using the geometry data type to visualize statistics data in SSMS.<h1>DBCC SHOW_STATISTICS</h1>
<p>While it may not be fun, running <a href="http://msdn.microsoft.com/en-us/library/ms174384.aspx">DBCC SHOW_STATISTICS</a> can tell you a lot about how the query optimizer might use a given index, or why certain queries are more susceptible to parameter sniffing.</p>
<p>If you are the visual type like myself you have probably taken the output from this command and pasted it into Excel to create a bar chart. This can be an extremely easy way to get a quick look at the statistics to see how evenly the column values are distributed in the B-Tree, but who wants to waste time opening Excel?</p>
<p>With the following script you can use SSMS's built-in drawing capabilities to create a histogram you can view right inside SSMS:</p>
<div class="codehilite"><pre><span></span><code><span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">TableName</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">128</span><span class="p">)</span><span class="w"></span>
<span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">StatisticsName</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">128</span><span class="p">)</span><span class="w"></span>
<span class="k">SET</span><span class="w"> </span><span class="o">@</span><span class="n">TableName</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="w"></span>
<span class="k">SET</span><span class="w"> </span><span class="o">@</span><span class="n">StatisticsName</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">''</span><span class="w"></span>
<span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">SQLCmd</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">2048</span><span class="p">)</span><span class="w"></span>
<span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">ScaleFactor</span><span class="w"> </span><span class="nb">NUMERIC</span><span class="p">(</span><span class="mi">8</span><span class="p">,</span><span class="mi">8</span><span class="p">)</span><span class="w"></span>
<span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">StatH</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="k">MAX</span><span class="p">)</span><span class="w"></span>
<span class="cm">/*</span>
<span class="cm">This script will draw a graphical histrogram for the given statistics.</span>
<span class="cm">Just provide a table name, and the name of the statistics you are</span>
<span class="cm">interested in and you will get a bar chart of the value distribution.</span>
<span class="cm">After executing the script, click on the "spatial results" tab to see the</span>
<span class="cm">chart.</span>
<span class="cm">*/</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="o">#</span><span class="n">histogram</span><span class="w"></span>
<span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">ID</span><span class="w"> </span><span class="nb">INT</span><span class="w"> </span><span class="k">IDENTITY</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="p">[</span><span class="n">RANGE_HI_KEY</span><span class="p">]</span><span class="w"> </span><span class="n">SQL_VARIANT</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">[</span><span class="n">RANGE_ROWS</span><span class="p">]</span><span class="w"> </span><span class="n">SQL_VARIANT</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">[</span><span class="n">EQ_ROWS</span><span class="p">]</span><span class="w"> </span><span class="n">SQL_VARIANT</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">[</span><span class="n">DISTINCT_RANGE_ROWS</span><span class="p">]</span><span class="w"> </span><span class="n">SQL_VARIANT</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">[</span><span class="n">AVG_RANGE_ROWS</span><span class="p">]</span><span class="w"> </span><span class="n">SQL_VARIANT</span><span class="w"></span>
<span class="p">)</span><span class="w"></span>
<span class="k">SET</span><span class="w"> </span><span class="o">@</span><span class="n">SQLCmd</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'DBCC SHOW_STATISTICS ("'</span><span class="o">+</span><span class="w"> </span><span class="o">@</span><span class="n">TableName</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s1">'","'</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="o">@</span><span class="n">StatisticsName</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s1">'") WITH HISTOGRAM'</span><span class="w"></span>
<span class="w"> </span><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="o">#</span><span class="n">histogram</span><span class="w"> </span><span class="k">EXEC</span><span class="p">(</span><span class="o">@</span><span class="n">SQLCmd</span><span class="p">);</span><span class="w"></span>
<span class="c1">--== To keep things visible I scale the RANGE_ROWS value</span>
<span class="c1">--== down if needed</span>
<span class="k">SET</span><span class="w"> </span><span class="o">@</span><span class="n">ScaleFactor</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="k">CASE</span><span class="w"> </span><span class="k">WHEN</span><span class="w"> </span><span class="k">MAX</span><span class="p">(</span><span class="k">CAST</span><span class="p">(</span><span class="n">RANGE_ROWS</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">BIGINT</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="k">CAST</span><span class="p">(</span><span class="n">EQ_ROWS</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">BIGINT</span><span class="p">))</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="k">THEN</span><span class="w"> </span><span class="mi">100</span><span class="p">.</span><span class="mi">0</span><span class="o">/</span><span class="p">(</span><span class="k">MAX</span><span class="p">(</span><span class="k">CAST</span><span class="p">(</span><span class="n">RANGE_ROWS</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">BIGINT</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="k">CAST</span><span class="p">(</span><span class="n">EQ_ROWS</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">BIGINT</span><span class="p">)))</span><span class="w"> </span><span class="k">ELSE</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="k">END</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="o">#</span><span class="n">histogram</span><span class="p">)</span><span class="w"></span>
<span class="c1">--== This constructs an enormous string of coordinates, one</span>
<span class="c1">--== set per shape:</span>
<span class="k">SET</span><span class="w"> </span><span class="o">@</span><span class="n">StatH</span><span class="w"> </span><span class="o">=</span><span class="w"></span>
<span class="n">STUFF</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="s1">',(('</span><span class="w"> </span><span class="o">+</span><span class="w"></span>
<span class="k">CAST</span><span class="p">(</span><span class="w"> </span><span class="n">ID</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">30</span><span class="p">)</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s1">' 0,'</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="c1">-- Bottom left</span>
<span class="k">CAST</span><span class="p">(</span><span class="w"> </span><span class="n">ID</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">30</span><span class="p">)</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s1">' '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="k">CAST</span><span class="p">(</span><span class="w"> </span><span class="k">CAST</span><span class="p">(</span><span class="n">RANGE_ROWS</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">BIGINT</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="k">CAST</span><span class="p">(</span><span class="n">EQ_ROWS</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">BIGINT</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="o">@</span><span class="n">ScaleFactor</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">30</span><span class="p">)</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s1">','</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="c1">-- Top Left</span>
<span class="k">CAST</span><span class="p">(</span><span class="w"> </span><span class="n">ID</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">.</span><span class="mi">75</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">30</span><span class="p">)</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s1">' '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="k">CAST</span><span class="p">(</span><span class="w"> </span><span class="k">CAST</span><span class="p">(</span><span class="n">RANGE_ROWS</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">BIGINT</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="k">CAST</span><span class="p">(</span><span class="n">EQ_ROWS</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">BIGINT</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="o">@</span><span class="n">ScaleFactor</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">30</span><span class="p">)</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s1">','</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="c1">-- Top Right</span>
<span class="k">CAST</span><span class="p">(</span><span class="w"> </span><span class="n">ID</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="p">.</span><span class="mi">75</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">30</span><span class="p">)</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s1">' 0,'</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="c1">--Bottom Right</span>
<span class="k">CAST</span><span class="p">(</span><span class="w"> </span><span class="n">ID</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">30</span><span class="p">)</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s1">' 0))'</span><span class="w"> </span><span class="c1">-- Back to the start, bottom left</span>
<span class="k">FROM</span><span class="w"> </span><span class="o">#</span><span class="n">histogram</span><span class="w"></span>
<span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">ID</span><span class="w"></span>
<span class="k">FOR</span><span class="w"> </span><span class="n">XML</span><span class="w"> </span><span class="n">PATH</span><span class="p">(</span><span class="s1">''</span><span class="p">)),</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"> </span><span class="s1">''</span><span class="p">);</span><span class="w"></span>
<span class="c1">--== MULTIPOLYGON allows us to draw multiple shapes from</span>
<span class="c1">--== a single string</span>
<span class="k">SELECT</span><span class="w"> </span><span class="n">geometry</span><span class="p">::</span><span class="n">STGeomFromText</span><span class="p">(</span><span class="w"> </span><span class="s1">'MULTIPOLYGON('</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="o">@</span><span class="n">StatH</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s1">')'</span><span class="p">,</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">GraphData</span><span class="p">;</span><span class="w"></span>
<span class="c1">--== Dumping the raw histogram data as well</span>
<span class="k">SELECT</span><span class="w"></span>
<span class="w"> </span><span class="n">ID</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">RANGE_HI_KEY</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">RANGE_ROWS</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">EQ_ROWS</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="o">#</span><span class="n">histogram</span><span class="p">;</span><span class="w"></span>
<span class="k">DROP</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="o">#</span><span class="n">histogram</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>To use the above query simply input the table and statistics you are interested in seeing and execute it, the graphical histogram will appear in the "Spatial Results" tab. For this example we are going to use the AdventureWorks2012 database and take a look at the statistics for the IX_SalesOrderDetail_ProductID on the Sales.SalesOrderDetail table:</p>
<p><img alt="Statistic Histogram" src="/img/SSMS_Statistics-04.png" /></p>
<p>In the image above you can see the "Spatial Results" tab at the top, and the graphical view of the histogram in the results area. Each line in the chart represents a step in the histogram. Using this quick visual tool you can eaily see how "balanced" your histogram is. In cases where it is highly unbalanced, like in the example, queries written that use compare a parameter value to the value in this column could be highly susceptible to poor parameter sniffing.</p>
<h1>So how does this all work?</h1>
<p>The Spatial Results tab shows geometric (spatial) data. Spatial data is great for storing things like map data, but in the end it's just a collection of connected points on a plane. What the above script is doing is simply plotting out a line for each step in the histogram and drawing it as a narrow box. When all the boxes are arranged next to eachother we end up with a bar chart.</p>
<h1>Further Resources</h1>
<p>Statistics are an interesting and important component of SQL Server, below are some resources to help you better understand statistics as well as learn more about displaying spatial data in SSMS.</p>
<p><a href="http://www.sqlpassion.at/archive/2014/01/28/inside-the-statistics-histogram-density-vector/">Inside the Statistics Histogram & Density Vector</a> @ www.sqlpassion.at/</p>
<p><a href="https://www.simple-talk.com/sql/learn-sql-server/statistics-in-sql-server/">Statistics in SQL Server</a> @ www.simple-talk.com</p>
<p><a href="http://sqlmag.com/t-sql/generating-charts-and-drawings-sql-server-management-studio">Generating Charts and Drawings in SQL Server Management Studio</a> @ www.sqlmag.com</p>
<p><a href="http://www.brentozar.com/archive/2013/06/the-elephant-and-the-mouse-or-parameter-sniffing-in-sql-server/">The Elephant and the Mouse, or, Parameter Sniffing in SQL Server</a> @ www.brentozar.com</p>wanted: a mentor2015-02-15mentorIn search of a mentor.<p>Mark Wilkinson, a budding DBA in Raleigh North Carolina, is looking for a skilled mentor to help him shape his future.</p>
<p>Qualified candidates must be able to provide guidance in some of the following areas:</p>
<ul>
<li>Public speaking</li>
<li>Writing for both blogs and books</li>
<li>Managing teams</li>
<li>Starting a business</li>
<li>Time management</li>
<li>Mentoring</li>
</ul>
<p>This is a 2 month position with optional email follow-ups after the term has ended. Works hours are flexible. This is a non-paid position, you will also not qualify for any benefits ( medical, health, dental, etc. ). You will however, get to enjoy the feeling of knowing that you have contributed to the future of an individual that gives back to the community when he can, and will likely go on to mentor future generations.</p>
<p>Founded in 1982, in Ypslianti Michigan, Mark Wilkinson is a "jack-of-all-trades" DBA. He has interests in all things development, technology, and databases. Outside of technology his interests include music, food, and playing various imaginary games with his children.</p>
<p>As a mentor you will get to see Mark work towards his goals of becoming a senior level DBA, becoming an authoritative voice in the SQL community, and someday, founding a non-profit to train children and teens in technological fields.</p>
<p>If you wish to apply for this position, please contact Mark Wilkinson at <a href="mailto:mark@m82labs.com">mark@m82labs.com</a>.</p>
<p><sup>* This post is a response to a post by Paul Randal: <a href="http://www.sqlskills.com/blogs/paul/want-mentored/">Want to be mentored by me?</a></sup></p>exists optimizations2015-02-24exists-optimizationsWhat's the best way to write an EXISTS clause?<p>If you've done any amount of SQL development, you've probably seen an <code>EXISTS</code> clause. It might have been part of an <code>IF</code> statement, or a <code>WHERE</code> clause, but I'm sure you've seen it, and maybe used it yourself. One thing you'll notice is that all developers have their own way of writing it. Some developers swear by <code>... EXISTS( SELECT 1 ...</code>, another might use <code>... EXISTS( SELECT TOP(1) * ...</code>. So what's the deal? Which method is best?</p>
<h1>The Setup</h1>
<p>First things first, lets see some of the different variations of the <code>EXISTS</code> clause, then we'll take a look at the execution plans and see what's going on behind the scenes.</p>
<blockquote>
<p>For these demo scripts we'll be using the AdventureWorks database</p>
</blockquote>
<p><strong>Example 1: SELECT *</strong></p>
<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span><span class="w"></span>
<span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">FirstName</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">LastName</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">Person</span><span class="p">.</span><span class="n">Person</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">a</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"> </span><span class="k">EXISTS</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"></span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">HumanResources</span><span class="p">.</span><span class="n">Employee</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">b</span><span class="w"></span>
<span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">BusinessEntityID</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">b</span><span class="p">.</span><span class="n">BusinessEntityID</span><span class="w"></span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">LastName</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'Johnson'</span><span class="w"></span>
<span class="w"> </span><span class="p">);</span><span class="w"></span>
</code></pre></div>
<p><strong>Example 2: SELECT TOP(1) *</strong></p>
<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span><span class="w"></span>
<span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">FirstName</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">LastName</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">Person</span><span class="p">.</span><span class="n">Person</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">a</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"> </span><span class="k">EXISTS</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="n">TOP</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="o">*</span><span class="w"></span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">HumanResources</span><span class="p">.</span><span class="n">Employee</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">b</span><span class="w"></span>
<span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">BusinessEntityID</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">b</span><span class="p">.</span><span class="n">BusinessEntityID</span><span class="w"></span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">LastName</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'Johnson'</span><span class="w"></span>
<span class="w"> </span><span class="p">);</span><span class="w"></span>
</code></pre></div>
<p><strong>Example 3: SELECT TOP(1) with defined column</strong></p>
<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span><span class="w"></span>
<span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">FirstName</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">LastName</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">Person</span><span class="p">.</span><span class="n">Person</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">a</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"> </span><span class="k">EXISTS</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="n">TOP</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="n">b</span><span class="p">.</span><span class="n">BusinessEntityID</span><span class="w"></span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">HumanResources</span><span class="p">.</span><span class="n">Employee</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">b</span><span class="w"></span>
<span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">BusinessEntityID</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">b</span><span class="p">.</span><span class="n">BusinessEntityID</span><span class="w"></span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">LastName</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'Johnson'</span><span class="w"></span>
<span class="w"> </span><span class="p">);</span><span class="w"></span>
</code></pre></div>
<p><strong>Example 4: SELECT 1</strong></p>
<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span><span class="w"></span>
<span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">FirstName</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">LastName</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">Person</span><span class="p">.</span><span class="n">Person</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">a</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"> </span><span class="k">EXISTS</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">HumanResources</span><span class="p">.</span><span class="n">Employee</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">b</span><span class="w"></span>
<span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">BusinessEntityID</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">b</span><span class="p">.</span><span class="n">BusinessEntityID</span><span class="w"></span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">LastName</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'Johnson'</span><span class="w"></span>
<span class="w"> </span><span class="p">);</span><span class="w"></span>
</code></pre></div>
<p>I'm sure there are more examples, but this will be enough for our purposes here. Let's take a look at the plans and io stats for these queries now and see how each performed.</p>
<h1>The Plans</h1>
<p>Below is the plan for the first query in our list. Nothing too exciting here:</p>
<p><img alt="Query Plan" src="/img/Exists-QueryPlan.png" /></p>
<h1>IO Stats</h1>
<table>
<thead>
<tr>
<th align="left">Scan Count</th>
<th align="left">Logical Reads</th>
<th align="left">Physical Reads</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">2</td>
<td align="left">6</td>
<td align="left">0</td>
</tr>
</tbody>
</table>
<p>The interesting thing happens when we take a look at the other query plans. Because of optimizations to the <code>EXISTS</code> clause, all 4 of these queries have an <em>identical</em> execution plan (other that the statement text included in the plan XML of course). From these tests it would seem that the optimizer doesn't care what you select, it's just looking at the query predicates to determine if something exists.</p>
<p>That sounds fine and good, but what if we make things a little more complicated? Here we are going to create a function that does some pointless work:</p>
<div class="codehilite"><pre><span></span><code><span class="k">ALTER</span><span class="w"> </span><span class="k">FUNCTION</span><span class="w"> </span><span class="n">fnWaitForSomeTime</span><span class="p">(</span><span class="w"> </span><span class="o">@</span><span class="n">num</span><span class="w"> </span><span class="nb">INT</span><span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="k">RETURNS</span><span class="w"> </span><span class="nb">INT</span><span class="w"></span>
<span class="k">AS</span><span class="w"></span>
<span class="k">BEGIN</span><span class="w"></span>
<span class="w"> </span><span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">count_num</span><span class="w"> </span><span class="nb">INT</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">loop_i</span><span class="w"> </span><span class="nb">INT</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">WHILE</span><span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="o">@</span><span class="n">loop_i</span><span class="w"> </span><span class="o"><</span><span class="w"> </span><span class="o">@</span><span class="n">num</span><span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">BEGIN</span><span class="w"></span>
<span class="w"> </span><span class="k">SET</span><span class="w"> </span><span class="o">@</span><span class="n">count_num</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="k">COUNT</span><span class="p">(</span><span class="o">*</span><span class="p">)</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">sys</span><span class="p">.</span><span class="n">columns</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="k">SET</span><span class="w"> </span><span class="o">@</span><span class="n">loop_i</span><span class="w"> </span><span class="o">+=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">END</span><span class="w"></span>
<span class="w"> </span><span class="k">RETURN</span><span class="w"> </span><span class="k">ABS</span><span class="p">(</span><span class="n">CHECKSUM</span><span class="p">(</span><span class="o">@</span><span class="n">count_num</span><span class="p">))</span><span class="w"></span>
<span class="k">END</span><span class="w"></span>
</code></pre></div>
<p>To test this function I ran the following:</p>
<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span><span class="w"> </span><span class="n">TOP</span><span class="p">(</span><span class="mi">1000</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">dbo</span><span class="p">.</span><span class="n">fnWaitForSomeTime</span><span class="p">(</span><span class="mi">10000</span><span class="p">)</span><span class="w"></span>
<span class="k">FROM</span><span class="w"></span>
<span class="w"> </span><span class="n">HumanResources</span><span class="p">.</span><span class="n">Employee</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">b</span><span class="w"></span>
</code></pre></div>
<p>On my system this ran for 60 seconds before I stopped it. So what happens if we add this to our <code>EXISTS</code>? What if we also add a CPU heavy <code>ORDER BY</code>?</p>
<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span><span class="w"></span>
<span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">FirstName</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">LastName</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">Person</span><span class="p">.</span><span class="n">Person</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">a</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"> </span><span class="k">EXISTS</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="n">TOP</span><span class="p">(</span><span class="mi">1000</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">dbo</span><span class="p">.</span><span class="n">fnWaitForSomeTime</span><span class="p">(</span><span class="mi">10000</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">FROM</span><span class="w"></span>
<span class="w"> </span><span class="n">HumanResources</span><span class="p">.</span><span class="n">Employee</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">b</span><span class="w"></span>
<span class="w"> </span><span class="k">WHERE</span><span class="w"></span>
<span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">BusinessEntityID</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">b</span><span class="p">.</span><span class="n">BusinessEntityID</span><span class="w"></span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">LastName</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'Johnson'</span><span class="w"></span>
<span class="w"> </span><span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"></span>
<span class="w"> </span><span class="n">HASHBYTES</span><span class="p">(</span><span class="s1">'MD5'</span><span class="p">,</span><span class="k">CAST</span><span class="p">(</span><span class="n">CHECKSUM</span><span class="p">(</span><span class="n">NEWID</span><span class="p">())</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">varbinary</span><span class="p">))</span><span class="w"></span>
<span class="w"> </span><span class="p">);</span><span class="w"></span>
</code></pre></div>
<p>Lets see the plan:</p>
<p><img alt="Query Plan" src="/img/Exists-QueryPlan.png" /></p>
<p>Notice anything familiar? Again we have the exact same plan. If you run this script yourself you'll see that it also takes the same amount of time to execute as the rest.</p>
<p>To take this a little further, lets try something we know shouldn't work (Thanks <a href="http://sqlstudies.com/about/">Kenneth Fischer</a> for this great example!):</p>
<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span><span class="w"></span>
<span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">FirstName</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">LastName</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">Person</span><span class="p">.</span><span class="n">Person</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">a</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"> </span><span class="k">EXISTS</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="mi">1</span><span class="o">/</span><span class="mi">0</span><span class="w"> </span><span class="c1">-- <---- What?!</span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">HumanResources</span><span class="p">.</span><span class="n">Employee</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">b</span><span class="w"></span>
<span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">BusinessEntityID</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">b</span><span class="p">.</span><span class="n">BusinessEntityID</span><span class="w"></span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">LastName</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'Johnson'</span><span class="w"></span>
<span class="w"> </span><span class="p">);</span><span class="w"></span>
</code></pre></div>
<p>Again, same plan, same execution time, and no error. This example just further illustrates that the optimizer doesn't care what you are selecting in an <code>EXISTS</code> clause, it only cares about the predicate.</p>
<h1>Conclusion</h1>
<p>We looked at a few variations of the <code>EXISTS</code> clause, and then saw how SQL Server handles them. Due to some nice optimizations SQL seems to do the least amount of work possible to check if something exists. This means that it is really up to you and your team to decide which is the most readable.</p>
<p>Personally I like to avoid <code>*</code> whenever I can, even if it doesn't cause a performance hit, so I tend to use the syntax in example 4 <code>... EXISTS( SELECT 1 ...</code>.</p>untangling dynamic sql2015-04-07untangle-dynamicDynamic SQL can still be readable.<p>There's a lot to be said for readability in code. Whenever you're writing code you should be thinking about readability. Performance is always important, and obviously
you want the code to be functionally correct, but if it's not readable and maintainable you might as well not even write it.</p>
<h1>Dynamic SQL</h1>
<p>Dynamic SQL can be challenging to read and challenging to write. I'm not sure I've ever met a developer that likes to maintain someone else's dynamic SQL. But there
is something you can do about it, and it might even make dynamic SQL a little bit fun.</p>
<p>Here is an example of some dynamic SQL code:</p>
<div class="codehilite"><pre><span></span><code><span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="k">SQL</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="k">MAX</span><span class="p">)</span><span class="w"></span>
<span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">ProductID</span><span class="w"> </span><span class="nb">INT</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">492</span><span class="w"></span>
<span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">MinQuantity</span><span class="w"> </span><span class="nb">INT</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">2</span><span class="w"></span>
<span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">MaxQuantity</span><span class="w"> </span><span class="nb">INT</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">15</span><span class="w"></span>
<span class="k">SET</span><span class="w"> </span><span class="o">@</span><span class="k">SQL</span><span class="w"> </span><span class="o">=</span><span class="w"></span>
<span class="s1">'SELECT Product.Name, Product.ProductNumber, '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">CHAR</span><span class="p">(</span><span class="mi">13</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"></span>
<span class="s1">' ProductInventory.LocationID, '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">CHAR</span><span class="p">(</span><span class="mi">13</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"></span>
<span class="s1">' ProductInventory.Quantity, '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">CHAR</span><span class="p">(</span><span class="mi">13</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"></span>
<span class="s1">''''</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="k">CONVERT</span><span class="p">(</span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">32</span><span class="p">),</span><span class="n">GETDATE</span><span class="p">(),</span><span class="mi">121</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="s1">''' AS ReportDate '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">CHAR</span><span class="p">(</span><span class="mi">13</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"></span>
<span class="s1">'FROM Production.Product '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">CHAR</span><span class="p">(</span><span class="mi">13</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"></span>
<span class="s1">'JOIN Production.ProductInventory '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">CHAR</span><span class="p">(</span><span class="mi">13</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"></span>
<span class="s1">' ON Product.ProductID = ProductInventory.ProductID '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">CHAR</span><span class="p">(</span><span class="mi">13</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"></span>
<span class="s1">'WHERE 1=1 '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">CHAR</span><span class="p">(</span><span class="mi">13</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"></span>
<span class="k">ISNULL</span><span class="p">(</span><span class="s1">' AND Product.ProductID = '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="k">CAST</span><span class="p">(</span><span class="o">@</span><span class="n">ProductID</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">512</span><span class="p">)</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">CHAR</span><span class="p">(</span><span class="mi">13</span><span class="p">),</span><span class="s1">''</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"></span>
<span class="k">ISNULL</span><span class="p">(</span><span class="s1">' AND ProductInventory.Quantity >= '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="k">CAST</span><span class="p">(</span><span class="w"> </span><span class="o">@</span><span class="n">MinQuantity</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">512</span><span class="p">)</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">CHAR</span><span class="p">(</span><span class="mi">13</span><span class="p">),</span><span class="w"> </span><span class="s1">''</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"></span>
<span class="k">ISNULL</span><span class="p">(</span><span class="s1">' AND ProductInventory.Quantity <= '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="k">CAST</span><span class="p">(</span><span class="w"> </span><span class="o">@</span><span class="n">MaxQuantity</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">512</span><span class="p">)</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="nb">CHAR</span><span class="p">(</span><span class="mi">13</span><span class="p">),</span><span class="w"> </span><span class="s1">''</span><span class="p">)</span><span class="w"></span>
<span class="k">EXEC</span><span class="p">(</span><span class="o">@</span><span class="k">SQL</span><span class="p">)</span><span class="w"></span>
</code></pre></div>
<p>This isn't likely on your list of queries to run in the future, but it illustrates a point. Dynamic SQL can be ugly. Now, if we follow a few simple formatting guidelines, and add a few instances of REPLACE, we get something much cleaner:</p>
<div class="codehilite"><pre><span></span><code><span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="k">SQL</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="k">MAX</span><span class="p">)</span><span class="w"></span>
<span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">ProductID</span><span class="w"> </span><span class="nb">INT</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">492</span><span class="w"></span>
<span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">MinQuantity</span><span class="w"> </span><span class="nb">INT</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">2</span><span class="w"></span>
<span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">MaxQuantity</span><span class="w"> </span><span class="nb">INT</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">15</span><span class="w"></span>
<span class="k">SET</span><span class="w"> </span><span class="o">@</span><span class="k">SQL</span><span class="w"> </span><span class="o">=</span><span class="w"></span>
<span class="k">REPLACE</span><span class="p">(</span><span class="k">REPLACE</span><span class="p">(</span><span class="k">REPLACE</span><span class="p">(</span><span class="k">REPLACE</span><span class="p">(</span><span class="k">REPLACE</span><span class="p">(</span><span class="w"></span>
<span class="c1">--=========================================================</span>
<span class="s1">'</span>
<span class="s1">SELECT</span>
<span class="s1"> Product.Name,</span>
<span class="s1"> Product.ProductNumber,</span>
<span class="s1"> ProductInventory.LocationID,</span>
<span class="s1"> ProductInventory.Quantity,</span>
<span class="s1"> "{{"{{Date"}}}}" AS ReportDate --<<-- We use a double-quote</span>
<span class="s1">FROM</span>
<span class="s1"> Production.Product</span>
<span class="s1"> JOIN Production.ProductInventory</span>
<span class="s1"> ON Product.ProductID = ProductInventory.ProductID</span>
<span class="s1">WHERE 1=1</span>
<span class="s1"> {{"{{ProductIDCondition"}}}}</span>
<span class="s1"> {{"{{QuantityGTCondition"}}}}</span>
<span class="s1"> {{"{{QuantityLTCondition"}}}}</span>
<span class="s1">'</span><span class="w"></span>
<span class="c1">--=========================================================</span>
<span class="p">,</span><span class="s1">'{{"{{ProductIDCondition"}}}}'</span><span class="p">,</span><span class="k">ISNULL</span><span class="p">(</span><span class="s1">' AND Product.ProductID = '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="k">CAST</span><span class="p">(</span><span class="o">@</span><span class="n">ProductID</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">512</span><span class="p">)</span><span class="w"> </span><span class="p">),</span><span class="s1">''</span><span class="p">))</span><span class="w"></span>
<span class="p">,</span><span class="s1">'{{"{{QuantityGTCondition"}}}}'</span><span class="p">,</span><span class="k">ISNULL</span><span class="p">(</span><span class="s1">' AND ProductInventory.Quantity >= '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="k">CAST</span><span class="p">(</span><span class="w"> </span><span class="o">@</span><span class="n">MinQuantity</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">512</span><span class="p">)</span><span class="w"> </span><span class="p">),</span><span class="w"> </span><span class="s1">''</span><span class="p">))</span><span class="w"></span>
<span class="p">,</span><span class="s1">'{{"{{QuantityLTCondition"}}}}'</span><span class="p">,</span><span class="k">ISNULL</span><span class="p">(</span><span class="s1">' AND ProductInventory.Quantity <= '</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="k">CAST</span><span class="p">(</span><span class="w"> </span><span class="o">@</span><span class="n">MaxQuantity</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">512</span><span class="p">)</span><span class="w"> </span><span class="p">),</span><span class="w"> </span><span class="s1">''</span><span class="p">))</span><span class="w"></span>
<span class="p">,</span><span class="s1">'{{"{{Date"}}}}'</span><span class="p">,</span><span class="k">CONVERT</span><span class="p">(</span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">32</span><span class="p">),</span><span class="n">GETDATE</span><span class="p">(),</span><span class="mi">121</span><span class="p">))</span><span class="w"></span>
<span class="p">,</span><span class="s1">'"'</span><span class="p">,</span><span class="s1">''''</span><span class="p">)</span><span class="w"> </span><span class="c1">--<-- Replace double-quote with a single-quote.</span>
<span class="k">EXEC</span><span class="p">(</span><span class="o">@</span><span class="k">SQL</span><span class="p">)</span><span class="w"></span>
</code></pre></div>
<h1>What We Did Here</h1>
<ol>
<li>Use a single block of SQL with place holders</li>
<li>Use double-braced place-holders:<ol>
<li>Field names/expressions {{fieldName}}</li>
<li>Variables/parameters {{@variableName}}</li>
</ol>
</li>
<li>Use double quotes in places you would normally use a single quote, then replace at the end. This can make things much easier to look at.</li>
</ol>
<p>The example above is doing a simple <code>SELECT</code> from AdventureWorks, which may not be the best use case for this ( Read about why you should be using <code>sp_executesql</code> here:
<a href="http://www.sommarskog.se/dynamic_sql.html">The Curse and Blessings of Dynamic SQL</a> ), but it can be great for maintenance/DBA scripts.
I am currently working on a method to script out job creation/modifications using TSQL (blog post coming), here is an example where this formatting really adds clarity to the code:</p>
<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span><span class="w"></span>
<span class="k">REPLACE</span><span class="p">(</span><span class="k">REPLACE</span><span class="p">(</span><span class="k">REPLACE</span><span class="p">(</span><span class="k">REPLACE</span><span class="p">(</span><span class="k">REPLACE</span><span class="p">(</span><span class="k">REPLACE</span><span class="p">(</span><span class="w"></span>
<span class="k">REPLACE</span><span class="p">(</span><span class="k">REPLACE</span><span class="p">(</span><span class="k">REPLACE</span><span class="p">(</span><span class="k">REPLACE</span><span class="p">(</span><span class="k">REPLACE</span><span class="p">(</span><span class="k">REPLACE</span><span class="p">(</span><span class="w"></span>
<span class="c1">--=========================================================</span>
<span class="s1">'</span>
<span class="s1">EXEC msdb.dbo.sp_update_schedule @schedule_id={{"{{scheduleID"}}}},</span>
<span class="s1"> @enabled={{"{{enabled"}}}},</span>
<span class="s1"> @freq_type={{"{{f_type"}}}},</span>
<span class="s1"> @freq_interval={{"{{f_interval"}}}},</span>
<span class="s1"> @freq_subday_type={{"{{f_subday_type"}}}},</span>
<span class="s1"> @freq_subday_interval={{"{{f_subday_interval"}}}},</span>
<span class="s1"> @freq_relative_interval={{"{{f_relative_interval"}}}},</span>
<span class="s1"> @freq_recurrence_factor={{"{{f_rec_factor"}}}},</span>
<span class="s1"> @active_start_date={{"{{a_start_date"}}}},</span>
<span class="s1"> @active_end_date={{"{{a_end_date"}}}},</span>
<span class="s1"> @active_start_time={{"{{a_start_time"}}}},</span>
<span class="s1"> @active_end_time={{"{{a_end_time"}}}};</span>
<span class="s1">'</span><span class="w"></span>
<span class="c1">--=========================================================</span>
<span class="p">,</span><span class="s1">'{{"{{scheduleID"}}}}'</span><span class="p">,</span><span class="n">schedule_id</span><span class="p">)</span><span class="w"></span>
<span class="p">,</span><span class="s1">'{{"{{enabled"}}}}'</span><span class="p">,</span><span class="n">enabled</span><span class="p">)</span><span class="w"></span>
<span class="p">,</span><span class="s1">'{{"{{f_type"}}}}'</span><span class="p">,</span><span class="n">freq_type</span><span class="p">)</span><span class="w"></span>
<span class="p">,</span><span class="s1">'{{"{{f_interval"}}}}'</span><span class="p">,</span><span class="n">freq_interval</span><span class="p">)</span><span class="w"></span>
<span class="p">,</span><span class="s1">'{{"{{f_subday_type"}}}}'</span><span class="p">,</span><span class="n">freq_subday_type</span><span class="p">)</span><span class="w"></span>
<span class="p">,</span><span class="s1">'{{"{{f_subday_interval"}}}}'</span><span class="p">,</span><span class="n">freq_subday_interval</span><span class="p">)</span><span class="w"></span>
<span class="p">,</span><span class="s1">'{{"{{f_relative_interval"}}}}'</span><span class="p">,</span><span class="n">freq_relative_interval</span><span class="p">)</span><span class="w"></span>
<span class="p">,</span><span class="s1">'{{"{{f_rec_factor"}}}}'</span><span class="p">,</span><span class="n">freq_recurrence_factor</span><span class="p">)</span><span class="w"></span>
<span class="p">,</span><span class="s1">'{{"{{a_start_date"}}}}'</span><span class="p">,</span><span class="n">active_start_date</span><span class="p">)</span><span class="w"></span>
<span class="p">,</span><span class="s1">'{{"{{a_end_date"}}}}'</span><span class="p">,</span><span class="n">active_end_date</span><span class="p">)</span><span class="w"></span>
<span class="p">,</span><span class="s1">'{{"{{a_start_time"}}}}'</span><span class="p">,</span><span class="n">active_start_time</span><span class="p">)</span><span class="w"></span>
<span class="p">,</span><span class="s1">'{{"{{a_end_time"}}}}'</span><span class="p">,</span><span class="n">active_end_time</span><span class="p">)</span><span class="w"></span>
<span class="k">FROM</span><span class="w"></span>
<span class="w"> </span><span class="n">msdb</span><span class="p">.</span><span class="n">dbo</span><span class="p">.</span><span class="n">sysschedules</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"></span>
<span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'MyTestSchedule'</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<h1>Conclusion</h1>
<p>Once you start using this format, it's hard to do it any other way. I was introduced to a version of this by the great Jeff Moden
( <a href="http://www.sqlservercentral.com/Authors/Articles/Jeff_Moden/80567/">See him @ SQLServerCentral</a> ) and I haven't looked back since.
Not only does this method make your code easier to read, it can also minimize the amount of time you spend hunting down unclosed single
quotes, missing spaces, and all the other stuff that makes dynamic SQL so much "fun" to work with.</p>reduce sql agent job overlaps2015-04-22reduce-overlapsCould overlapping jobs be slowing you down?<p>When you notice a performance issue on an instance, what is your first instinct? I would say the majority of DBAs would start looking at the user and application processes the were running against the instance. While it's a pretty safe bet to make, how do you know you're not a part of the problem?</p>
<p>When was the last time you looked at your SQL Agent jobs? Do you have any jobs with overlapping execution times? I'm not just talking about jobs using the same schedule, but jobs that overlap every tenth execution, or every fourth. If you have one or two instances to manage it can be a trivial task to check this out, there are even a few pieces of software that can help. But what if you manage 100+ instances, with 100 jobs each? In this post we'll go over a custom built solution developed just for such a case.</p>
<h1>Overview</h1>
<p>The general concept is straight forward: We will calculate and add a delay to the execution of each job in such a way that we get the least amount of overlapping executions. We do this by getting all job execution data for a 24 hour period and then we add a small (ever increasing) delay to each job until we reach the lowest level of overlaps possible. This delay information is then stored in a table on the instance and is used by a 'Delay' step added to each job to delay job execution by the amount specified.</p>
<h1>The Moving Parts</h1>
<p>There are a few moving parts to this solution:</p>
<ul>
<li>[JobDelay] table</li>
<li>[AddJobDelay] stored procedure</li>
<li>[GetJobData] stored procedure</li>
<li>Overlap Checker C# console app</li>
</ul>
<p><strong>JobDelay Table</strong>: This table stores the job name and an integer representing the number of seconds to delay each execution of the job.</p>
<p><strong>AddJobDelay Stored Procedure</strong>: This procedure loops through all jobs on the instance and adds a new 'Delay' step as the first step of the job. This step executes the following code:</p>
<div class="codehilite"><pre><span></span><code><span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">delay</span><span class="w"> </span><span class="nb">INT</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">NULL</span><span class="p">;</span><span class="w"></span>
<span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">waitfor</span><span class="w"> </span><span class="nb">CHAR</span><span class="p">(</span><span class="mi">8</span><span class="p">);</span><span class="w"></span>
<span class="k">SELECT</span><span class="w"> </span><span class="o">@</span><span class="n">delay</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">delay_sec</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">JobDelay</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"> </span><span class="n">job_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'[job name goes here]'</span><span class="w"></span>
<span class="k">SET</span><span class="w"> </span><span class="o">@</span><span class="n">waitfor</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">LEFT</span><span class="p">(</span><span class="n">DATEADD</span><span class="p">(</span><span class="k">second</span><span class="p">,</span><span class="k">ISNULL</span><span class="p">(</span><span class="o">@</span><span class="n">delay</span><span class="p">,</span><span class="mi">0</span><span class="p">),</span><span class="k">CAST</span><span class="p">(</span><span class="s1">'00:00:00'</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="k">TIME</span><span class="p">)),</span><span class="mi">8</span><span class="p">);</span><span class="w"></span>
<span class="n">WAITFOR</span><span class="w"> </span><span class="n">DELAY</span><span class="w"> </span><span class="o">@</span><span class="n">waitfor</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>Because I didn't want to have to keep track of job ids, this solution requires that you use unique job names. Which is arguably something you should be doing anyhow.</p>
<p><strong>GetJobData Stored Procedure</strong>: This procedure returns one row for every job execution scheduled for the next 24 hours. Currently it does not pay attention to the time of day that a job can run (coming in a future version) it just assumes all jobs execute on a 24 hour schedule. As an example, if a job executes every 15 minutes, you would see 96 rows for this job. Along with executions, this proc also determines the average duration for the job (minus the delay step), this is very important later in the process.</p>
<p><strong>Overlap Checker</strong>: This is a C# console app that takes the data from the <code>GetJobData</code> stored procedure and calculates the optimal delay for each job. Here is an overview of what this program is doing (with as little detail as I can muster):</p>
<ol>
<li>Get job information via `GetJobData` stored procedure</li>
<li>For each unique job in the result set:
<ul>
<li>Get all the job executions for the current job, including the start and estimated end time for each execution.</li>
<li>Get all execution times for all other jobs on the instance, getting the start and estimated end time, taking into account any delays that have already been calulcated.</li>
<li>For each execution of the current job, check for overlaps with the rest of the executions.</li>
<li>If there are any overlaps, store the overlap count and add a small delay to each execution of the current job.</li>
<li>Check for overlaps again, if there are fewer overlaps (but more than zero), store the delay temporarily and repeat the previous step.</li>
<li>Continue looping until we get to 0 overlaps, OR our delay has reached 50% of the interval between job executions. For example, if a job executes every 10 minutes, the delay would never be longer than 5 minutes. Because we are only storing the delay amount if the overlap count lowers, we should end up with the least amount of delay for the least amount of overlaps.</li>
</ul>
</li>
<li>Once all delays are calculated, they are inserted into the `JobDelay` table on the instance.</li>
</ol>
<h1>Assumptions/Limitations</h1>
<p>In order for this to work we have to operate under a few assumptions:
- Jobs don't need to execute <em>exactly</em> when they were defined to run
- There is currently enough job history data in <code>msdb</code> to be considered representative of a typical day
- We don't care about reducing overlaps of jobs that execute every minute, or every twelve hours.</p>
<p>Limitations:
- All jobs must be uniquely named
- Jobs with multiple schedules are currently ignored</p>
<h1>Outcome and Thoughts</h1>
<p>Overall this solution has worked great. We saw an overall CPU reduction and "smoothing" effect on CPU spikes. I did doubt the effects at first, but turning the delays off quickly proved me wrong. Most jobs never see a delay more than 10-100 seconds, and overall this process only takes ~1 minute to execute per instance.</p>
<p>There was a big focus on getting execution times down on the application portion of this because I wanted to be able to run it directly on the instance. The initial version of this solution relied on pure TSQL and took ~45 minutes to execute, subsequent iterations used PowerShell which got it down to ~15 miuntes, and finally C# which got the execution times down to 30-45 seconds on average. Because of the amount of looping that occurs, moving this into a C# app was the right thing to do.</p>
<h1>Future Plans</h1>
<p><strong>Exclusions</strong>: The case has not yet come up where we need a job excluded from getting a delay assigned to it, but it is bound to happen. To accommodate this I will eventually be adding an exclusions table to each instance. It will likely be replicated or kept in sync between instances in some other fashion (I'm all about centralized management).</p>
<p><strong>Move More Into The App</strong>: I may also look into moving even more of the processing work into the C# app, currently I am using some CTE magic to extrapolate the job execution times based on the start date and execution interval, but this could easily be done in the app, and with less CPU overhead.</p>
<p><strong>Multiple Schedule Support</strong>: Eventually I want to add support for jobs with multiple schedules. To be honest it was something I didn't think about until the end, and in our environment we don't have many jobs using multiple schedules.</p>
<p><strong>Support For Execution Periods</strong>: <s>If a job is set to run every 15 minutes between 9am and 5pm, the overlap checker still treats it like it executes every 15 minutes all day long. This isn't an issue in my current environment but I will be adding support for this in future versions.</s> <strong>THIS FUNCTIONALITY HAS BEEN ADDED</strong></p>
<p>Eventually this code will be incorporated into a bigger project to create a central job management system (similar features to the MSX/TSX framework currently available on EE, but not requiring EE). When that happens it will open up the option to run this across an entire environment, this would allow you to reduce overlap on IO/CPU/Network intensive jobs across all instances to reduce load on your SAN, and network. Stay tuned for updates!</p>
<h1>The Code</h1>
<p>I created a new GitHub repo for this project. All code is available there, along with a README that walks you through installation. Keep an eye on this repo for changes: <a href="https://github.com/m82labs/overlap_checker">https://github.com/m82labs/overlap_checker</a></p>retrieving deadlock graphs with powershell2015-06-09deadlock-graph-poshHow to retrieve and store deadlocks graphs using PowerShell.<p>Before we dive into some extended events and PowerShell fun I want to say thanks to Jes Borland (<a href="http://blogs.lessthandot.com/index.php/author/grrlgeek/">WWW</a>/<a href="https://twitter.com/grrl_geek">Twitter</a>) for hosting this months T-SQL Tuesday! If you are interested in learning more about T-SQL Tuesday, take a look at Jes's <a href="http://blogs.lessthandot.com/index.php/uncategorized/youre-invited-to-t-sql-tuesday-67-extended-events/">post</a>, and check out the <a href="https://twitter.com/hashtag/tsql2sday?f=realtime&src=hash">#tsqltuesday</a> hashtag on Twitter.</p>
<h1>System Health</h1>
<p>Extended Events can be intimidating to start working with. You have to choose which events you want to capture, how you want to store it, and how you want to filter it. Scrolling through the available event list alone can be a little overwhelming. Thankfully SQL Server ships with the handy <code>system_health</code> extended event (XE going forward) session already set up and running.</p>
<p>The system health XE session has an <strong>amazing</strong> amount of information in it. Do you have some long-waiting tasks? Check system health. Need CPU history? Check system health. Connection errors? Check system health. Sev 20+ errors? Check system health. There's a lot more information available in this XE session, but for now we are just going to focus on one: deadlocks.</p>
<h1>Viewing Deadlocks in System Health</h1>
<p>Before we query system health for deadlocks, let's take a look at it in SSMS. Getting there is pretty simple, once connected to an instance via the object explorer, just go to <em>Management</em> > <em>Extended Events</em> > <em>Sessions</em> > <em>system_health</em>:</p>
<p><img alt="System Health XE" src="/img/SSMS_SystemHealth.png" /></p>
<p>To view the session data, double-click on <em>package0.event_file</em>. Once it opens up you'll see the default layout, which include two columns <strong>name</strong> and <strong>timestamp</strong>. The <strong>name</strong> column contains the name of the event type that was captured, and the <strong>timestamp</strong> tells you when it happened, single-clicking on any of these events will show more detail at the bottom of the window:</p>
<p><img alt="System Health Event File" src="/img/SSMS_SystemHealth-01.png" /></p>
<p>In this case we found a deadlock graph, and as you can see the beginning of the deadlock XML is shown. If you now click on the 'Deadlock' tab, you can see the graphical representation of the deadlock:</p>
<p><img alt="System Health Event File" src="/img/SSMS_SystemHealth-02.png" /></p>
<p>When looking at your system health session using SSMS, make sure you also take advantage of it's filtering capabilites, you can filter on both the timestamp and event name. To set up a filter just click on the 'Filters...' button in the toolbar at the top of the window. Below you can see how I would set up a filter to just show deadlocks (this test system only has a single deadlock):</p>
<p><img alt="System Health Event File" src="/img/SSMS_SystemHealth-03.png" /></p>
<p>I highly recommend taking a look around at some other events and see what you can find. There is a lot of documentation out there on system health, so find something interesting in yours and google it. You'll be surprised how much information is available.</p>
<h1>Querying Deadlocks From System Health</h1>
<p>Writing queries to get data out of extended events sessions can get complicated fast. To keep things simple we are going to use the <strong>ring buffer</strong> target for our queries. This isn't always the best approach, but for our purposes it will work. If you were going to set something up to grab deadlock graphs on a regular basis in a production environment I would recommend, without hesitation, that you do the work of querying the <strong>file target</strong>. To learn more about this process, check the resources section at the end of this post for a great article from Jonathan Kehayias.</p>
<p>Here is a simple script to grab deadlock graphs:</p>
<div class="codehilite"><pre><span></span><code><span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">system_health</span><span class="w"> </span><span class="n">XML</span><span class="w"></span>
<span class="c1">-- Copy ring buffer into an XML variable</span>
<span class="k">SELECT</span><span class="w"></span>
<span class="w"> </span><span class="o">@</span><span class="n">system_health</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">CAST</span><span class="p">(</span><span class="n">xet</span><span class="p">.</span><span class="n">target_data</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">XML</span><span class="p">)</span><span class="w"></span>
<span class="k">FROM</span><span class="w"></span>
<span class="w"> </span><span class="n">sys</span><span class="p">.</span><span class="n">dm_xe_session_targets</span><span class="w"> </span><span class="n">xet</span><span class="w"></span>
<span class="w"> </span><span class="k">INNER</span><span class="w"> </span><span class="k">JOIN</span><span class="w"> </span><span class="n">sys</span><span class="p">.</span><span class="n">dm_xe_sessions</span><span class="w"> </span><span class="n">xe</span><span class="w"></span>
<span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="n">xe</span><span class="p">.</span><span class="n">address</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">xet</span><span class="p">.</span><span class="n">event_session_address</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"></span>
<span class="w"> </span><span class="n">xe</span><span class="p">.</span><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'system_health'</span><span class="w"></span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">xet</span><span class="p">.</span><span class="n">target_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'ring_buffer'</span><span class="w"></span>
<span class="c1">-- Get the time stamp and deadlock graph</span>
<span class="k">SELECT</span><span class="w"></span>
<span class="w"> </span><span class="n">xed</span><span class="p">.</span><span class="n">value</span><span class="p">(</span><span class="s1">'@timestamp'</span><span class="p">,</span><span class="s1">'datetime'</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="n">xed</span><span class="p">.</span><span class="n">query</span><span class="p">(</span><span class="s1">'.'</span><span class="p">)</span><span class="w"></span>
<span class="k">FROM</span><span class="w"></span>
<span class="w"> </span><span class="o">@</span><span class="n">system_health</span><span class="p">.</span><span class="n">nodes</span><span class="p">(</span><span class="s1">'RingBufferTarget/event[@name="xml_deadlock_report"]'</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">XEventData</span><span class="p">(</span><span class="n">xed</span><span class="p">)</span><span class="w"></span>
</code></pre></div>
<p>We have a few things going here:</p>
<ul>
<li>First we copy data from the ring buffer directly into an XML variable. This simple operation can speed up XML parsing tremendously</li>
<li>Next we use the <code>nodes()</code> method of the XML variable to extract all event elements with an event name of <code>xml_deadlock_report</code></li>
</ul>
<p>The <code>nodes()</code> method is an important thing to understand when dealing with XML data like that found in extended events sessions. Basically <code>nodes()</code> is a function that returns XML elements as row data. What this means is that when you select <code>FROM</code> it, each row is an independent lump of XML data. In the above query our call to <code>nodes()</code> will return one row per <code>xml_deadlock_report</code> event that has been captured.</p>
<p>To extract useful information we are using the <code>value</code> and <code>query</code> methods. <code>value</code> takes the name of an attribute (timestamp in this case) or the path to a sub-element, and converts the value into the datatype specified. <code>query</code> takes an XQuery and returns an XML object, in our case we are specifying '.' which just means 'return everything'.</p>
<p>If you have never written XQuery before I suggest downloading <a href="https://xpathvisualizer.codeplex.com/">XPath Visualizer</a>. With this tool you can open an XML deadlock graph (or a query plan for that matter) and immediately see the effects of an XQuery:</p>
<p><img alt="System Health - XPath Visualizer" src="/img/SSMS_SystemHealth-04.png" /></p>
<h1>Querying with PowerShell</h1>
<p>So with all of that out of the way, lets do what we set out to do from the start. We already have the query to get our XML deadlock graphs, now we are going to combine it with a little PowerShell.</p>
<div class="codehilite"><pre><span></span><code><span class="c"># ==== Deadlock Test Script ---------------------------------------------------</span>
<span class="no">[string]</span><span class="nv">$instance</span> <span class="p">=</span> <span class="s1">'localhost'</span>
<span class="c"># ====-------------------------------------------------------------------------</span>
<span class="c"># ==== Set up the script ------------------------------------------------------</span>
<span class="nv">$get_deadlocks</span> <span class="p">=</span> <span class="s1">'</span>
<span class="s1">DECLARE @system_health XML</span>
<span class="s1">-- Copy ring buffer into an XML variable</span>
<span class="s1">SELECT</span>
<span class="s1"> @system_health = CAST(xet.target_data AS XML)</span>
<span class="s1">FROM</span>
<span class="s1"> sys.dm_xe_session_targets xet</span>
<span class="s1"> INNER JOIN sys.dm_xe_sessions xe</span>
<span class="s1"> ON xe.address = xet.event_session_address</span>
<span class="s1">WHERE</span>
<span class="s1"> xe.name = ''system_health''</span>
<span class="s1"> AND xet.target_name = ''ring_buffer''</span>
<span class="s1">-- Get the time stamp and deadlock graph</span>
<span class="s1">SELECT</span>
<span class="s1"> xed.value(''@timestamp'',''datetime'') AS timestamp,</span>
<span class="s1"> xed.query(''.'') As deadlockReport</span>
<span class="s1">FROM</span>
<span class="s1"> @system_health.nodes(''RingBufferTarget/event[@name="xml_deadlock_report"]'') AS XEventData(xed);</span>
<span class="s1">'</span>
<span class="c"># ==== Run the prepared script, output to object ------------------------------</span>
<span class="nv">$results</span> <span class="p">=</span> <span class="nb">Invoke-Sqlcmd</span> <span class="n">-ServerInstance</span> <span class="nv">$instance</span> <span class="n">-Query</span> <span class="nv">$get_deadlocks</span>
<span class="c"># ==== Output to screen -------------------------------------------------------</span>
<span class="nv">$results</span> <span class="p">|</span> <span class="nb">Format-Table</span> <span class="n">-AutoSize</span>
<span class="c"># ====-------------------------------------------------------------------------</span>
</code></pre></div>
<p>At this point you have your timestamp and the XML deadlock graph. Now all we need to do is write this out to a file. With PowerShell this is pretty trivial using the <code>Out-File</code> cmdlet. We simply loop through the results, create a unique file name, then save it.</p>
<div class="codehilite"><pre><span></span><code><span class="p">...</span>
<span class="k">Foreach</span><span class="p">(</span> <span class="nv">$result</span> <span class="k">IN</span> <span class="nv">$results</span> <span class="p">)</span> <span class="p">{</span>
<span class="c"># Generate a output file path</span>
<span class="no">[string]</span><span class="nv">$timestring</span> <span class="p">=</span> <span class="s1">'{0:yyyyMMdd-HHmmssfff}'</span> <span class="o">-f</span> <span class="no">[datetime]</span><span class="nv">$result</span><span class="p">.</span><span class="n">timestamp</span>
<span class="no">[string]</span><span class="nv">$outfile</span> <span class="p">=</span> <span class="s2">"$outpath$instance_$timestring-"</span> <span class="p">+</span> <span class="p">((</span><span class="nb">Get-Random</span><span class="p">)</span> <span class="p">%</span> <span class="n">100</span><span class="p">).</span><span class="n">ToString</span><span class="p">()</span> <span class="p">+</span> <span class="s2">"_deadlock.xdl"</span>
<span class="nb">Write-Host</span> <span class="s2">"Writing file: $outfile"</span>
<span class="k">try</span> <span class="p">{</span>
<span class="nb">Out-File</span> <span class="n">-InputObject</span> <span class="p">(</span><span class="nv">$result</span><span class="p">.</span><span class="n">deadlockReport</span><span class="p">)</span> <span class="n">-FilePath</span> <span class="nv">$outfile</span>
<span class="p">}</span> <span class="k">catch</span> <span class="p">{</span>
<span class="nb">Write-Host</span> <span class="s2">"Writing file $outfile failed: $_"</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">...</span>
</code></pre></div>
<p>Cool, now if you go check you output directory (we used C:\temp), you should see some deadlock graph <code>XDL</code> files in there. Double-clicking on one should open it in SSMS for you to view, but in our case it just returns an error:</p>
<p><img alt="System Health - XPath Visualizer" src="/img/SSMS_SystemHealth-05.png" /></p>
<p>If you open your deadlock graph in a text editor, you'll see that the XML ends kind of abruptly:</p>
<div class="codehilite"><pre><span></span><code>...
<span class="nt"></frame></executionStack><inputbuf></span>
Proc [Database Id = 11 Object Id = 665274371] <span class="nt"></inputbuf></process></process-list><resource-list><keylock</span> <span class="na">hobtid=</span>
</code></pre></div>
<p>So it looks like our file was truncated. But why would that be? The answer lies in the <code>Invoke-SqlCmd</code> cmdlet. By default it only allows a maximum of 4000 characters per column to be returned. Luckily we can override this behavior using the <code>-MaxCharLength</code> parameter. I was unable to find a max value for this parameter, but I find using a very large number works just fine:</p>
<div class="codehilite"><pre><span></span><code><span class="p">...</span>
<span class="c"># ==== Run the prepared script, output to object ------------------------------</span>
<span class="nv">$results</span> <span class="p">=</span> <span class="nb">Invoke-Sqlcmd</span> <span class="n">-ServerInstance</span> <span class="nv">$instance</span> <span class="n">-Query</span> <span class="nv">$get_deadlocks</span> <span class="n">-MaxCharLength</span> <span class="n">1000000</span>
<span class="p">...</span>
</code></pre></div>
<p>Now when we run our script and try to open one of our deadlock files we should be able to view both the graph and XML data in SSMS.</p>
<p>Here is our final script:</p>
<div class="codehilite"><pre><span></span><code><span class="c"># ==== Deadlock Test Script ---------------------------------------------------</span>
<span class="no">[string]</span><span class="nv">$instance</span> <span class="p">=</span> <span class="s1">'instance_name'</span>
<span class="no">[string]</span><span class="nv">$outpath</span> <span class="p">=</span> <span class="s1">'C:\temp\'</span>
<span class="c"># ====-------------------------------------------------------------------------</span>
<span class="c"># ==== Set up the script, then replace the variables --------------------------</span>
<span class="nv">$get_deadlocks</span> <span class="p">=</span> <span class="s1">'</span>
<span class="s1">DECLARE @system_health XML</span>
<span class="s1">-- Copy ring buffer into an XML variable</span>
<span class="s1">SELECT</span>
<span class="s1"> @system_health = CAST(xet.target_data AS XML)</span>
<span class="s1">FROM</span>
<span class="s1"> sys.dm_xe_session_targets xet</span>
<span class="s1"> INNER JOIN sys.dm_xe_sessions xe</span>
<span class="s1"> ON xe.address = xet.event_session_address</span>
<span class="s1">WHERE</span>
<span class="s1"> xe.name = ''system_health''</span>
<span class="s1"> AND xet.target_name = ''ring_buffer''</span>
<span class="s1">-- Get the time stamp and deadlock graph</span>
<span class="s1">SELECT</span>
<span class="s1"> xed.value(''@timestamp'',''datetime'') AS timestamp,</span>
<span class="s1"> xed.query(''.'') As deadlockReport</span>
<span class="s1">FROM</span>
<span class="s1"> @system_health.nodes(''RingBufferTarget/event[@name="xml_deadlock_report"]'') AS XEventData(xed);</span>
<span class="s1">'</span>
<span class="c"># ==== Run the prepared script, output to object ------------------------------</span>
<span class="nv">$results</span> <span class="p">=</span> <span class="nb">Invoke-Sqlcmd</span> <span class="n">-ServerInstance</span> <span class="nv">$instance</span> <span class="n">-Query</span> <span class="nv">$get_deadlocks</span> <span class="n">-MaxCharLength</span> <span class="n">1000000</span>
<span class="c"># ==== Output to screen -------------------------------------------------------</span>
<span class="nv">$results</span> <span class="p">|</span> <span class="nb">Format-Table</span> <span class="n">-AutoSize</span>
<span class="c"># ====-------------------------------------------------------------------------</span>
<span class="k">Foreach</span><span class="p">(</span> <span class="nv">$result</span> <span class="k">IN</span> <span class="nv">$results</span> <span class="p">)</span> <span class="p">{</span>
<span class="c"># Generate a output file path</span>
<span class="no">[string]</span><span class="nv">$timestring</span> <span class="p">=</span> <span class="s1">'{0:yyyyMMdd-HHmmssfff}'</span> <span class="o">-f</span> <span class="no">[datetime]</span><span class="nv">$result</span><span class="p">.</span><span class="n">timestamp</span>
<span class="no">[string]</span><span class="nv">$outfile</span> <span class="p">=</span> <span class="s2">"$outpath$instance_$timestring-"</span> <span class="p">+</span> <span class="p">((</span><span class="nb">Get-Random</span><span class="p">)</span> <span class="p">%</span> <span class="n">100</span><span class="p">).</span><span class="n">ToString</span><span class="p">()</span> <span class="p">+</span> <span class="s2">"_deadlock.xdl"</span>
<span class="nb">Write-Host</span> <span class="s2">"Writing file: $outfile"</span>
<span class="k">try</span> <span class="p">{</span>
<span class="nb">Out-File</span> <span class="n">-InputObject</span> <span class="p">(</span><span class="nv">$result</span><span class="p">.</span><span class="n">deadlockReport</span><span class="p">)</span> <span class="n">-FilePath</span> <span class="nv">$outfile</span>
<span class="p">}</span> <span class="k">catch</span> <span class="p">{</span>
<span class="nb">Write-Host</span> <span class="s2">"Writing file $outfile failed: $_"</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<h1>Final Thoughts</h1>
<p>This post was inspired by situations I ran into when trying to get XML data off of SQL Server and onto a file share. Hopefully this post helps you avoid some of the issues I ran into. We covered the basics here; there is still a lot of work that could be done to improve this and make it something you might want to run in a production environment. If you wanted to use this in your environment I would suggest looking into the following:</p>
<ul>
<li><strong>Using parameters in PowerShell</strong> This would allow you to pass in an instance name and output file path.</li>
<li><strong>Using the asynchronous file target</strong> This is a much more efficient method of getting data out of system health, especially if you use the offset method described in the sqlskills.com article below.</li>
</ul>
<h1>Resources</h1>
<p><a href="https://www.sqlskills.com/blogs/jonathan/an-xevent-a-day-6-of-31-targets-week-asynchronous_file_target/">An XEvent a Day (6 of 31) – Targets Week – asynchronous_file_target</a> @ www.sqlskills.com</p>
<p><a href="https://technet.microsoft.com/en-us/magazine/jj554301.aspx">Windows PowerShell: Defining Parameters</a> @ technet.microsoft.com</p>
<p><a href="https://msdn.microsoft.com/en-us/library/ms191474%28v=sql.110%29.aspx">query() Method (xml Data Type)</a> @ msdn.microsoft.com</p>
<p><a href="https://msdn.microsoft.com/en-us/library/ms178030%28v=sql.110%29.aspx">value() Method (xml Data Type)</a> @ msdn.microsoft.com</p>mariadb backups for the sql server dba2015-07-02mariadb-backupLearn to do backups, the MariaDB way.<p>As a DBA your most important job is making sure you don't lose your data in the case of catastrophic failure. Why should that be any different when your data is stored in a system you don't know very well?</p>
<p>With the plethora of database systems available today most DBAs will find themselves supporting more than just SQL Server. This will be the first in a series of posts that will explore backup and restore options in a variety of relational and non-relational systems from the perspective of a SQL Server DBA. While this series will not be comprehensive by any means (there are 10+ engine options for MariaDB alone) I will try to cover the most common use cases for each system.</p>
<h1>Prerequisites</h1>
<p>This post, and really the whole series, is going to assume that you have some experience on the Linux command line and that you have access to a lab environment where you can follow along. While some of the systems in this series <strong>can</strong> run on Windows, they weren't typically designed with it in mind. If you have not used Linux I highly suggest you spin up a VM and give it a try. All the command examples given are going to be executed on an Ubuntu Linux box, but the commands should work in any distribution.</p>
<h1>MariaDB and the XtraDB Storage Engine</h1>
<p>MariaDB is a DBMS designed to be a drop-in replacement for MySQL, even using the same binary and config file names. It was developed as a fork of MySQL by some of the original developers after concerns arose when MySQL was purchased by Oracle. For the most part, MariaDB is almost 100% compatible with existing MySQL code.</p>
<p><sup>For more details on the incompatibilities, check out the <a href="https://mariadb.com/kb/en/mariadb/mariadb-vs-mysql-compatibility/">MariaDB website</a>.</sup></p>
<p>MariaDB can use a variety of storage engines depending on what you are using it for. For example, if you are looking for a traditional ACID-compliant RDBMS like SQL Server you can use the XtraDB engine, if you are looking for an in-memory database the MEMORY storage engine might be a good fit, if you need to query data from CSV, XML, or JSON files you can use the CONNECT engine . Depending on your needs you can even mix it up and use different engines for different tables. All of these options can make it an extremely flexible solution, and at a price of $FREE, it's worth giving it a try.</p>
<p>For the rest of this post we are going to focus on the default storage engine for MariaDB, XtraDB. XtraDB is a high-performance, backwards-compatible, fork of InnoDB developed by <a href="https://www.percona.com/software/percona-server/percona-xtradb">Percona</a>. It's important to note that XtraDB is compatible with both MySQL and MariaDB, so this post can work for either.</p>
<h1>XtraBackup and Innobackupex</h1>
<p>Xtrabackup is a tool developed by Percona to perform non-locking backups of your database. It's a little confusing at first, but Xtrabackup is only part of the story, what you will be spending the most time with is Innobackupex. Innobackupex is a Perl wrapper script that calls Xtrabackup and adds a lot of nice functionality like automatically time-stamping your backups (more on this later). There are more options for backing up your data than just Xtrabackup/Innobackupex, but for the SQL Server DBA this option should seem the most familiar.</p>
<blockquote>
<p>There are many other options when taking backups on a MariaDB or MySQL instance. I chose to cover xtrabackup in this post because it will likely be the most familiar for the typical SQL Server DBA, it gives you a lot of flexible backup options, and it allows you to take online backups with a minimal amount of setup.</p>
</blockquote>
<h2>Before You Run a Backup</h2>
<p>Before you run your first backup you need to ensure you have the following:</p>
<h3>1. Add backup system users to the <code>mysql</code> group.</h3>
<p>Any users that need to perform backups should be added to this group. To add the <code>m82labs</code> user to it (you'll want to add your own user for this example) execute the following:</p>
<div class="codehilite"><pre><span></span><code>$ sudo usermod -G mysql -a m82labs
</code></pre></div>
<p>You'll also need to make some changes to the permissions on your MariaDB data file directories for this to work properly (here we assume your data directory is <code>/var/lib/mysql</code>):</p>
<div class="codehilite"><pre><span></span><code>$ find /var/lib/mysql -type d -exec sudo chmod <span class="m">750</span> <span class="o">{}</span> <span class="se">\;</span>
</code></pre></div>
<p>These changes will give the <code>mysql</code> user full access to the files, users in the <code>mysql</code> group get read and execute (read allows us to read the file for backup, execute allows us to traverse the directories), and anyone else gets no permissions at all.</p>
<blockquote>
<p>Linux permissons can be confusing at first, I highly recommend reading up on using <code>chmod</code> to get a good understanding of how to set permissions. Check the <a href="#resources"><em>Resources</em></a> section at the bottom of the post for more information on using <code>chmod</code>.</p>
</blockquote>
<p>We will also need to make some changes to how MariaDB create directories for new databases so we don't have to manually change permissions each time we create a new database. To do this you will need to modify the <code>/etc/init.d/mysql</code> file and add the following lines to the top of the file just below the <code>INIT INFO</code> header block:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">UMASK_DIR</span><span class="o">=</span><span class="m">424</span> <span class="c1"># = Evaluates to 750</span>
<span class="nb">export</span> UMASK_DIR
</code></pre></div>
<p>After this change you will need restart the database service:
:::bash
$ sudo service mysql restart</p>
<p>Then log out and log back in for the group membership changes to register.</p>
<h3>2. A backup directory that is owned by the <code>mysql</code> user.</h3>
<p>Make a new directory and change the permissions and ownership. This ensures that the <code>mysqld</code> service can write to the directory, and any user in the <code>mysql</code> group can also manipulate the contents of the directory.</p>
<div class="codehilite"><pre><span></span><code>$ sudo mkdir /opt/backups
$ sudo chown -R mysql:mysql /opt/backups
$ sudo chmod -R <span class="m">770</span> /opt/backups
</code></pre></div>
<h3>3. A database user with the correct permissions.</h3>
<p>Typically you should create a dedicated user for backups. Here we'll assume you named it <code>backup</code>. Since this user is going to be handling your backups only, it should be safe to assume the user will only connect from the local machine.</p>
<p>In MariaDB and MySQL a user isn't just identified by a username, but also a hostname. As an example, you could create four different <code>backup</code> users with four different passwords as long as you specified that they all connect from different hosts. Though not recommended in this case, putting a <code>%</code> in the hostname field will allow the user to connect from any computer.</p>
<p>To create the user connect to your MariaDB instance and execute the following statements:</p>
<div class="codehilite"><pre><span></span><code><span class="k">CREATE</span><span class="w"> </span><span class="k">USER</span><span class="w"> </span><span class="s1">'backup'</span><span class="o">@</span><span class="s1">'localhost'</span><span class="p">;</span><span class="w"></span>
<span class="k">SET</span><span class="w"> </span><span class="n">PASSWORD</span><span class="w"> </span><span class="k">FOR</span><span class="w"> </span><span class="s1">'backup'</span><span class="o">@</span><span class="s1">'localhost'</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">PASSWORD</span><span class="p">(</span><span class="s1">'SuperSecret'</span><span class="p">);</span><span class="w"></span>
</code></pre></div>
<p>This creates a user named <code>backup</code> that can only connect from <code>localhost</code> with a password of <code>SuperSecret</code>. This account will need the following permissions:</p>
<ul>
<li><code>RELOAD</code></li>
<li><code>LOCK TABLES</code></li>
<li><code>REPLICATION CLIENT</code></li>
<li><code>CREATE TABLESPACE</code></li>
<li><code>PROCESS</code></li>
<li><code>SUPER</code></li>
</ul>
<p>To <code>GRANT</code> the required permissions, execute the following:</p>
<div class="codehilite"><pre><span></span><code><span class="k">GRANT</span><span class="w"> </span><span class="n">RELOAD</span><span class="p">,</span><span class="k">LOCK</span><span class="w"> </span><span class="n">TABLES</span><span class="p">,</span><span class="n">REPLICATION</span><span class="w"> </span><span class="n">CLIENT</span><span class="p">,</span><span class="k">CREATE</span><span class="w"> </span><span class="n">TABLESPACE</span><span class="p">,</span><span class="n">PROCESS</span><span class="p">,</span><span class="n">SUPER</span><span class="w"></span>
<span class="k">ON</span><span class="w"> </span><span class="o">*</span><span class="p">.</span><span class="o">*</span><span class="w"> </span><span class="k">TO</span><span class="w"> </span><span class="s1">'backup'</span><span class="o">@</span><span class="s1">'localhost'</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>Now that all of this is set up we can move onto creating some backups.</p>
<h2>Anatomy of a Backup</h2>
<p>Taking backups with <code>Innobackupex</code> is pretty straight forward:</p>
<div class="codehilite"><pre><span></span><code>$ innobackupex --user<span class="o">=</span>backup --password<span class="o">=</span><span class="s1">'SuperSecret'</span> /opt/backups
</code></pre></div>
<p>This command is simply connecting to your MariaDB instance with the supplied username and password, making a backup, and storing it in the directory you specified. You <em>can</em> supply a host and port if you like, but by default it will attempt to connect to localhost using the port specified in the servers <code>/etc/mysql/my.cnf</code> file. If no such file exists it will check the <code>$MYSQL_TCP_PORT</code> environment variable, the <code>/etc/services</code> file (see more about this file <a href="http://linux.about.com/cs/linux101/g/slshetcslshserv.htm">here</a>), or if all else fails, the default port of <code>3306</code>.</p>
<p>After running the above command you should see a bunch of output on the screen. Assuming you have no errors, lets go check out the <code>/opt/backups</code> directory and see what we have.</p>
<div class="codehilite"><pre><span></span><code>$ ls -la /opt/backups
</code></pre></div>
<blockquote>
<p>If you do have errors, run through all the steps again and make sure you didn't miss anything. Typically errors during a backup are related to permissions.</p>
</blockquote>
<p>One of the first things you'll notice about backups from <code>innobackupex</code> is that your backup is not just a single file, it is an entire directory named with a time stamp. If you run an <code>ls</code> on that directory you will see several files and sub-directories in it. One of the interesting things about this backup is that it contains ALL the information needed to restore the backup.</p>
<p>There is no concept of <code>msdb</code> in the MariaDB world. All of the information that would normally be found in the various backup related DMVs is available in the files located in your new backup directory. Here is a brief description of what you will find here (visit the <a href="https://www.percona.com/doc/percona-xtrabackup/2.1/xtrabackup-files.html">Percona website</a> for more detail):</p>
<ul>
<li>
<p><strong><em>backup-my.cnf:</em></strong> After a backup is taken it still needs to be "prepared" (more on that later) before it can be used. This config file is used to start a small DB instance to prepare your backup for a restore.</p>
</li>
<li>
<p><strong><em>xtrabackup_checkpoints:</em></strong> This file contains information about the type of backup you took along with the range of LSNs involved in the backup.</p>
</li>
<li>
<p><strong><em>xtrabackup_binary:</em></strong> A copy of the xtrabackup binary used to perform the backup.</p>
</li>
<li>
<p><strong><em>xtrabackup_logfile:</em></strong> This is the equivalent of the transaction log. There are no transaction log-only backups in MariaDB, the transaction log is automatically included when you run a backup.</p>
</li>
</ul>
<h2>Preparing Your Backups</h2>
<p>Before we get into the details of the various backup types, it is very important that we discuss the process of 'preparing' a backup. As we touched on before, preparing a backup gets the backup ready for restoring.</p>
<p>When you take a backup it cannot copy all the required files into the backup directory at the exact same moment in time, some files get copied a few seconds after others. Running a <code>prepare</code> on the backup will get all files point-in-time consistent with each other.</p>
<p>The prepare also accomplishes another important thing, it tests your backup. If something is wrong with your backup, the prepare process will fail. Because of this, a lot of people like to run a prepare right after they take a backup. This is a great idea, but as we will discuss below, it's not always an option.</p>
<h2>Backup Types</h2>
<p>You have a lot of options when backing up your MariaDB data. Innobackupex allows you to take full backups, incremental, partial, and compact backups. Each have pros and cons and in the end you have a lot of options for a very flexible backup schedule.</p>
<h2>Full Backup</h2>
<p>This is the simplest of the backups, and is still required if you decide to take incremental backups. One thing to note about the full backup is that it is a full <strong><em>instance</em></strong> backup, not just a single DB. That being said, when you take a full backup you can still restore a single database or table from that full backup, it is just a bit of work. This will be discussed in more detail later on.</p>
<h3>Taking a Full Backup</h3>
<p>Earlier in this post we saw the command for a full backup, but we'll repeat it here:</p>
<div class="codehilite"><pre><span></span><code>$ innobackupex --user<span class="o">=</span>backup --password<span class="o">=</span><span class="s1">'SuperSecret'</span> /opt/backups
</code></pre></div>
<p>This will create a directory named with the current timestamp in the <code>/opt/backups/</code> directory. You could take another full immediately after this one is done and it would create another timestamped directory for you.</p>
<h3>Preparing a Full Backup</h3>
<p>As we mentioned earlier, the backup has to be prepared before it can be restored. To prepare a backup you simply run the following command:</p>
<div class="codehilite"><pre><span></span><code>$ innobackupex --user<span class="o">=</span>backup --password<span class="o">=</span><span class="s1">'SuperSecret'</span> --apply-log --use-memory<span class="o">=</span>1G /opt/backups/YourBackupDirectory
</code></pre></div>
<p>The <code>--use-memory</code> parameter is optional, but if you give the prepare process more memory it can definitely speed things up. You'll have to play with this option to see what works best for your system.</p>
<blockquote>
<p>Note: This step should be skipped if you plan on doing incremental backups based off of this full backup. In that case you could copy the backup to a different location and attempt a prepare there.</p>
</blockquote>
<h3>Restoring a Full Backup</h3>
<p>After preparing the backup we can do an actual restore. Since a full backup is a backup of all databases on the instance, the database service needs to be stopped before a restore can begin and all existing data in the data directory needs to be deleted. I would highly recommend creating a "staging" area of sorts to hold the original contents of your data directory <strong>before</strong> you delete it. This way, if something goes wrong with the restore, you can always just copy your original files back over.</p>
<div class="codehilite"><pre><span></span><code>$ sudo mkdir /opt/mysql.temp
$ sudo chown mysql:mysql /opt/mysql.temp
$ sudo chmod <span class="m">770</span> /opt/mysql.temp
</code></pre></div>
<p>To stop the database service and restore the backup:</p>
<div class="codehilite"><pre><span></span><code>$ sudo service mysql stop
$ sudo mv /var/lib/mysql/* /opt/mysql.temp/
$ innobackupex --copy-back /opt/backups/YourBackupDirectory
</code></pre></div>
<p>This will copy your database files to the empty data directory, now we need to fix ownership issues (the files will be owned by the user that ran the restore) and restart the database service:</p>
<div class="codehilite"><pre><span></span><code>$ sudo chown -R mysql:mysql /var/lib/mysql
$ sudo service mysql start
</code></pre></div>
<p>With any luck you should now have a functional instance based on the backup you restored. Make sure you remember to delete everything from your "staging" area after the restore is complete and well-tested.</p>
<h2>Incremental Backup</h2>
<p>Xtrabackup supports <strong>true incremental backups</strong>. Unlike SQL Servers differential backups, which store the changes made since the last full backup, incremental backups store the changes made since the last incremental backup.</p>
<h3>Taking an Incremental Backup</h3>
<p>The first step in taking incremental backups is to take a full backup to base your incremental on. For our purposes lets assume we already have a full backup from our steps above at the following path: <code>/opt/backups/2015-06-28_13-42-58/</code></p>
<p>To take an incremental you would issue the following command:</p>
<div class="codehilite"><pre><span></span><code>$ innobackupex --user<span class="o">=</span>backup --password<span class="o">=</span><span class="s1">'SuperSecret'</span> --incremental /opt/backups --incremental-basedir<span class="o">=</span>/opt/backups/2015-06-28_13-42-58/
</code></pre></div>
<p>The key to this command is the <code>--incremental-basedir</code> parameter. This tells innobackupex to base the incremental off of whatever backup is in the directory specified. If you recall from earlier, each backup folder contains a file named <code>xtrabackup_checkpoints</code>, this file tells innobackupex which LSN to start it's incremental backup at.</p>
<p>After running this command you will have just another timestamped directory in your backups directory. To find your incremental backups you could execute a command like this:</p>
<div class="codehilite"><pre><span></span><code>$ find /opt/backups -type f <span class="p">|</span> xargs grep incremental
</code></pre></div>
<p>You should see output similar to this:
:::bash
/opt/backups/2015-06-28_15-10-37/xtrabackup_checkpoints:backup_type = incremental</p>
<p>From here you can continue taking incremental backups, each one based off of the previous incremental, for example the next incremental would have a <code>--incremental-basedir</code> of <code>/opt/backups/2015-06-28_15-10-37/</code>. There is also an option available to base your incremental simply on a known LSN. We aren't really going to go into any detail here, but it would do roughly the same operations as basing it off of an existing backup, except you have to manually specify the starting LSN via the <code>--incremental-lsn</code> parameter.</p>
<h3>Preparing an Incremental Backup</h3>
<p>Preparing an incremental backup is a little different than preparing a full backup. Earlier we mentioned that if you choose to take incremental backups you <strong>cannot</strong> prepare your backups right after you take them. The reason for this is that when preparing an incremental backup, all incremental backups need to be applied to the base full backup before anything can be prepared.</p>
<p>This process is very similar to the concept of restoring differential backups and transaction logs in SQL Server, you don't want your subsequent restores to go through recovery until all backups are applied.</p>
<p>To begin we need to apply our logs to the base full backup we initially created:</p>
<div class="codehilite"><pre><span></span><code>$ innobackupex --user<span class="o">=</span>backup --password<span class="o">=</span><span class="s1">'SuperSecret'</span> --apply-log --redo-only /opt/backups/2015-06-28_13-42-58/
</code></pre></div>
<p>This brings the full backup to a state where all logs have been applied, but uncommitted transactions have <strong>NOT</strong> been rolled back.</p>
<p>Now we start applying out incremental backups:</p>
<div class="codehilite"><pre><span></span><code>$ innobackupex --user<span class="o">=</span>backup --password<span class="o">=</span><span class="s1">'SuperSecret'</span> --apply-log --redo-only /opt/backups/2015-06-28_13-42-58/ --incremental-dir<span class="o">=</span>/opt/backups/2015-06-28_15-10-37/
</code></pre></div>
<p>If we had more incremental backups we would continue to execute this command, changing the <code>--incremental-dir</code> parameter to apply each incremental backup <strong><em>in the order they were taken</em></strong>. When you get to your final incremental to apply, you need to omit the <code>--redo-only</code> option, but if you forget, it's no big deal, the server will handle it automatically.</p>
<p>Once all of your incremental backups are applied, we need to prepare the final backup. This statement needs to be run on the directory the base full backup resides in, the one we based all of our incrementals on:</p>
<div class="codehilite"><pre><span></span><code>$ innobackupex --user<span class="o">=</span>backup --password<span class="o">=</span><span class="s1">'SuperSecret'</span> --apply-log --use-memory<span class="o">=</span>1G /opt/backups/2015-06-28_13-42-58/
</code></pre></div>
<p>Now just as before we can go through the steps of restoring a full backup.</p>
<p>Stop the service, remove the old data files, and copy the new data files:</p>
<div class="codehilite"><pre><span></span><code>$ sudo service mysql stop
$ sudo rm -rf /var/lib/mysql/*
$ innobackupex --copy-back /opt/backups/YourBackupDirectory
</code></pre></div>
<p>Now we fix any ownership issues and restart the service:</p>
<div class="codehilite"><pre><span></span><code>$ sudo chown -R mysql:mysql /var/lib/mysql
$ sudo service mysql start
</code></pre></div>
<h2>Compact Backups</h2>
<p>The compact backup isn't so much a <em>type</em> of backup as an option when you are taking backups. A compact backup does not include any secondary index pages, a secondary index in MariaDB is analogous to a non-clustered index in SQL Server.</p>
<p>It's important to understand that a compact backup <strong>does</strong> include the metadata needed to recreate the indexes, but it <strong>does not</strong> include the actual index pages themselves. What that means is that you can reduce your backup size while still retaining the ability to rebuild your secondary indexes at a later time.</p>
<p>The compact backup can potentially be much smaller than a standard backup depending on the number of secondary indexes you have. This has two obvious benefits: less space required per backup, and less time to actually perform the backup. Like all things in life and computers though, you are trading space and time <strong>NOW</strong> for space and time <strong>LATER</strong>. When you need to restore a compact backup, you will need to rebuild the indexes, which will take time and consume additional space.</p>
<p>You can take a compact backup by adding the <code>--compact</code> option to either a full or incremental backup. When you create a compact backup the <code>compact</code> flag will be set to <code>1</code> in the <code>xtrabackup_checkpoints</code> file in the backup directory. If you want to find you compact backups you can run this command:</p>
<div class="codehilite"><pre><span></span><code>$ find /opt/backups -type f <span class="p">|</span> xargs grep -B <span class="m">4</span> <span class="s1">'compact = 1'</span>
</code></pre></div>
<p>You should see output similar to this:</p>
<div class="codehilite"><pre><span></span><code>/opt/backups/2015-06-29_16-10-04/xtrabackup_checkpoints-backup_type <span class="o">=</span> full-backuped
/opt/backups/2015-06-29_16-10-04/xtrabackup_checkpoints-from_lsn <span class="o">=</span> <span class="m">0</span>
/opt/backups/2015-06-29_16-10-04/xtrabackup_checkpoints-to_lsn <span class="o">=</span> <span class="m">1644288</span>
/opt/backups/2015-06-29_16-10-04/xtrabackup_checkpoints-last_lsn <span class="o">=</span> <span class="m">1644288</span>
/opt/backups/2015-06-29_16-10-04/xtrabackup_checkpoints:compact <span class="o">=</span> <span class="m">1</span>
--
/opt/backups/2015-06-29_16-11-11/xtrabackup_checkpoints-backup_type <span class="o">=</span> incremental
/opt/backups/2015-06-29_16-11-11/xtrabackup_checkpoints-from_lsn <span class="o">=</span> <span class="m">1644288</span>
/opt/backups/2015-06-29_16-11-11/xtrabackup_checkpoints-to_lsn <span class="o">=</span> <span class="m">1644288</span>
/opt/backups/2015-06-29_16-11-11/xtrabackup_checkpoints-last_lsn <span class="o">=</span> <span class="m">1644288</span>
/opt/backups/2015-06-29_16-11-11/xtrabackup_checkpoints:compact <span class="o">=</span> <span class="m">1</span>
</code></pre></div>
<h3>Preparing and Restoring a Compact Backup</h3>
<p>Preparing and restoring a compact backup is simple. To prepare the backup you would prepare the backup as usual but include the <code>--rebuild-indexes</code> option.</p>
<p>In the case of a full backup there is only a single step involved in preparing the backup, so the <code>--rebuild-indexes</code> option would be included in that step. When preparing incremental backups the <code>--rebuild-indexes</code> option should only be included in the final step of the prepare, once you are applying the final incremental backup to the full, base, backup.</p>
<p>Once the index rebuilds are complete, and the logs are applied, your backup will be full-size, as if you have taken a normal (non-compact) backup, so make sure you have enough space before you start the prepare process. At this point you can restore the backup as usual via the <code>--copy-back</code> option discussed earlier.</p>
<blockquote>
<p>Compact can be a great way to save space and reduce the amount of time it takes to create a backup. While compact is great,</p>
</blockquote>
<h2>Table or Database-Specific Restores</h2>
<p>Now what if you only wanted to restore a single database? Well, that gets a bit tougher. If you want to do this level of restore you need to make sure the <code>innodb_file_per_table</code> option is enabled. If you are using MariaDB 5.5 or greater this is on by default.</p>
<blockquote>
<p>Technically you cannot restore a single database from a prepared backup, but what you can do is restore each of the tables in the database, effectively restoring the entire database.</p>
</blockquote>
<h3>The Restore Process</h3>
<p>For the examples below lets assume we want to restore a database creatively named <code>mydatabase</code>. We will walk through the steps to restore a single table named <code>OneofMyTables</code>. To restore a specific database, you would need to go through this process for each table in that database.</p>
<p>The first step in this process is to export all (yes all) of the tables from your prepared backup:</p>
<div class="codehilite"><pre><span></span><code>$ innobackupex --apply-log --export /opt/backups/YourBackupDirectory
</code></pre></div>
<p>Now, in the database you want to restore the tables to you will have to drop the old table and recreate it:</p>
<div class="codehilite"><pre><span></span><code><span class="n">USE</span><span class="w"> </span><span class="n">mydatabase</span><span class="p">;</span><span class="w"></span>
<span class="k">DROP</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">OneOfMyTables</span><span class="p">;</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">OneOfMyTables</span><span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="n">MyColumn</span><span class="w"> </span><span class="nb">INT</span><span class="w"> </span><span class="p">);</span><span class="w"></span>
</code></pre></div>
<p>Then we need to discard the tablespace for each table:</p>
<div class="codehilite"><pre><span></span><code><span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">OneOfMyTables</span><span class="w"> </span><span class="n">DISCARD</span><span class="w"> </span><span class="n">TABLESPACE</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>Now we need to remove the existing data files and copy the <code>OneOfMyTables.ibd</code> and <code>OneOfMyTables.exp</code> files from our backup to our data directory:
:::bash
$ cd /var/lib/mysql/mydatabase/
$ sudo rm -rf OneOfMyTables.*
$ sudo cp /opt/backups/YourBackupDirectory/mydatabase/OneOfMyTables.exp .
$ sudo cp /opt/backups/YourBackupDirectory/mydatabase/OneOfMyTables.ibd .</p>
<p>Since you are executing these commands as a user, we'll need to change the file ownership back so MariaDB can read the files. This step can wait until all of your table export files have been copied over:</p>
<div class="codehilite"><pre><span></span><code>$ sudo chown -R mysql:mysql /var/lib/mysql/mydatabase/
</code></pre></div>
<p>Executing this statement from within the <code>mydatabase</code> database:</p>
<div class="codehilite"><pre><span></span><code><span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">mydatabase</span><span class="p">.</span><span class="n">OneOfMyTables</span><span class="w"> </span><span class="n">IMPORT</span><span class="w"> </span><span class="n">TABLESPACE</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>Now you should be all set. The table should be available to query and it should contain the data from the backup.</p>
<h3>Foreign Key Considerations</h3>
<p>If you are dealing with tables that have foreign key constraints, it is up to you to make sure all related tables are imported in a consistent state. If you are importing a table that is used by other tables for foreign key look-ups, you will need to disable foreign key checks before you drop the original table:</p>
<div class="codehilite"><pre><span></span><code><span class="k">SET</span><span class="w"> </span><span class="n">FOREIGN_KEY_CHECKS</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>Once the table has been completely imported you can turn your foreign key checks back on:</p>
<div class="codehilite"><pre><span></span><code><span class="k">SET</span><span class="w"> </span><span class="n">FOREIGN_KEY_CHECKS</span><span class="o">=</span><span class="mi">1</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>From this point <strong>forward</strong> your foreign key constraints will function as usual, but there is a caveat here. If the data you imported violates a foreign key constraint you will not know until you try to manipulate the record that violates the constraint. MariaDB has no built-in mechanism to recheck your foreign key constraints. Interestingly, some adventurous users have written scripts and tools to take care of this, I am not going to cover those scripts here but I will include a link the <a href="#resources">resources</a> section.</p>
<h3>Thoughts on Table Imports</h3>
<p>Table imports are kind of a clunky process, but it's still a pretty cool feature. It's worth going over this a few times with some test data to get a good understanding of the steps involved. There are some good scripts people have written to attempt to automate this process, but it's always useful learning how to do it by hand.</p>
<h2>Partial Backup</h2>
<p>Partial backups are a special type of backup that allow you to backup specific tables or databases instead of backing up everything. While this sounds great, it has it's drawbacks.</p>
<h3>Taking a Partial Backup</h3>
<p>There are three ways you can take a partial backup. Each method is defined by a different option:</p>
<ul>
<li>
<p><strong><em>--include:</em></strong> This is the most flexible and the most complex to use. The <code>--include</code> option identifies which tables to backup via regular expression (regex). In our <code>mydatabase.OneOfMyTables</code> example from above we might specify a regex of <code>^mydatabase[.]OneOfMyTables$</code>. If we wanted to grab the whole database we might use <code>^mydatabase[.]*</code>. If you are familiar with regex you can see the flexibility here, if you are not I suggest reading up on it.</p>
</li>
<li>
<p><strong><em>--tables-file:</em></strong> This option takes a text file as an argument, this text file contains a fully qualified table per line (database.table).</p>
</li>
<li>
<p><strong><em>--databases:</em></strong> This option takes a space separated list of database names and/or fully qualified table names OR a text file containing a list with one database and/or fully qualified table per line.</p>
</li>
</ul>
<p>All of these options will result in a backup that looks pretty much like all the other backups you have seen so far, the only difference is that instead of seeing one directory per database, you will see one directory per database that was specified and one database directory per table that was specified. For example, if you used the <code>--databases</code> option and passed it <code>TestDB TestDB01.SomeTable</code> you would see a directory for <code>TestDB</code> and a directory for <code>TestDB01</code>.</p>
<h3>Preparing and Restoring a Partial Backup</h3>
<p>Unfortunately restoring a partial backup can really only be done via the export and import table process we discussed above in the <em>Table or Database-Specific Restores</em> section. The only other option is to restore ONLY the partial backup and wipe out the rest of your data. This could be useful in some limited situations.</p>
<p>As mentioned, you <strong><em>can</em></strong> restore a partial backup using the same methods you would use to restore a full backup, but the following must be true:</p>
<ul>
<li>The <code>mysql</code> database was included in the partial backup</li>
<li>Your data directory must be empty</li>
</ul>
<p>If those conditions are met, you can restore a partial backup by following the steps for preparing and restoring (via <code>--copy-back</code>) a full backup.</p>
<h1>Final Thoughts</h1>
<p>Overall you have a lot of options when using Perconas Xtrabackup software, but as I was told by a user on the #mysql IRC channel "it's not exactly polished".
This article really just scratches the surface when it comes to your backup and restore options, Xtrabackup is just one piece of software to accomplish this.</p>
<p>If you are interested in diving in a little deeper you'll find a lot people doing things like maintaining a replica slave specifically for executing backups, other people are using LVM or SAN snapshots to take backups, while some are still able to use the <code>mysqldump</code> utility to literally dump the sql scripts needed to rebuild the schema of the instance and populate the tables with data.</p>
<p>Whatever your needs are you should be able to find a backup process that works for you in the MariaDB/MySQL world. Compared to SQL Server you might find that some of it takes a little more work, or a little more time to get used to, but in the end you should still be more than able to effectively backup the data you have been charged with protecting.</p>
<h1>Coming Soon</h1>
<p>We covered a lot in this post, but there is still more to cover! Keep an eye out for the the second post in this series where we will dive into some more advanced backup and restore options, including streaming compressed backups and encrypted backups.</p>
<h1>Thanks</h1>
<p>Special thanks to Anthony E. Nocentino @ <a href="http://centinosystems.com">centinosystems.com</a> for proof-reading this post and offering some suggestions! Anthony is the Enterprise Architect at Centino Systems, you can find him on twitter: <a href="https://twitter.com/nocentino">@nocentino</a></p>
<h1>Resources</h1>
<p><a name="resources"></a></p>
<p>Below is a list of websites and other places you can find more information on MariaDB, MySQL, Percona, and Linux:</p>
<ul>
<li>#mysql, #maria, and #percona on <a href="https://freenode.net">Freenode</a> IRC. #mysql being the most active channel</li>
<li><a href="https://www.percona.com/doc/percona-xtrabackup/2.1/innobackupex/innobackupex_script.html">Percona Innobackupex Documentation</a> @ percona.com</li>
<li><a href="https://mariadb.com/kb/en/">MariaDB Knowledge Base</a> @ mariadb.com</li>
<li><a href="http://stackoverflow.com/questions/2250775/force-innodb-to-recheck-foreign-keys-on-a-table-tables">Force InnoDB to recheck foreign keys on a table/tables?</a> @ stackoverflow.com</li>
<li><a href="http://www.opsschool.org/en/latest/unix_101.html">Unix Fundamentals 101</a> @ opsschool.org</li>
<li><a href="https://www.digitalocean.com/community/tutorials/an-introduction-to-linux-permissions">An Introduction to Linux Permissions</a> @ digitalocean.com</li>
<li><a href="https://en.wikipedia.org/wiki/Chmod">chmod</a> @ en.wikipedia.org</li>
</ul>i'm still here2016-07-02still-hereI haven't died, I just have more kids.<p>I'm still here. The only updates I have made to the blog in the past year have been updating my 'about' page to note the new addition to our family (which is a great thing!). My last blog post was published two weeks before my daughter was born, and now that her first birthday is coming up I feel like I'm in a good place to start writing again. Writing technical blog posts is such a departure from my day-to-day work that it can be hard to get "in the zone", especially with a new baby (now fully mobile toddler) around the house. </p>
<p>That being said, I did get a lot of things done during my break from writing for my blog. I wanted to post some of those things here in case it's of interest to anyone out there. </p>
<h1>PassCard</h1>
<p>PassCard is a secure password system I came up with years ago that I just now got around to putting up on GitHub. PassCard creates a keycard you can carry around and use to create complex passwords using simple words. It's reusable, and as long as you remember your generation password, you can can easily regenerate and print copies of your card if your first one goes through the wash. Check out the project on GitHub and get some more details on this project: <a href="https://github.com/m82labs/passcard">https://github.com/m82labs/passcard</a></p>
<h1>Murrow</h1>
<p>I am a huge Linux nerd and love the older web technologies like RSS. These two interests, combined with a love of Python, lead to Murrow. Murrow is a console-based RSS aggregator written in Python 3. This is an educational project, but it has been a lot of fun and I use it every day. Murrow has some neat features (for a console app) like read-time estimates based on your actual read times of past articles, and <a href="http://getpocket.com">pocket</a> integration. Check out the project here: <a href="https://github.com/m82labs/murrow">https://github.com/m82labs/murrow</a></p>
<h1>Elasticsearch</h1>
<p>No, I haven't contributed anything to the great Elasticsearch project, but I have been using it a lot. Expect to see future posts making a lot of use of Elasticsearch and Kibana to visualize performance data from SQL Server and create dashboards. </p>
<h1>What's Next?</h1>
<p>News posts! I expect to finish up a post on Resource Governor in the next week or less, as well as a longer post outlining some tips and tricks for managing multiple SQL Server instances. Even though it has been a year, I do plan to at least write a follow-up to my popular MariaDB backup post covering some of the advanced options and usage.</p>monitoring resource usage with resource governor2016-09-30resource-governorKeep track of resource usage per user or application using Resource Governor.<p>When Resource Governor was first introduced with SQL Server 2008 you could use it to put limits on CPU usage, memory usage, and concurrent requests. In later editions Microsoft also added the ability to limit disk IO, place a hard cap on CPU usage, and control NUMA node affinity. Resource Governor is a great way to ensure no single application or user can completely starve another application or user of their SQL Server resources, but thats not what we are going to focus on today.</p>
<p>In this post we'll go over the basics of getting resource pools and workload groups set up, writing an effective classifier function to spearate your workload into these groups, and finally we'll talk about using the built-in DMVs to monitor resource usage per resource group and why this is such a cool feature.</p>
<p>In the examples below we will be creating a simple Resource Governor configuration for an environment where we have two teams executing requests, each team has two different applications that make these requests. As stated above, while we will discuss some of the surface-level basics of Resource Governor configuration, the goal of this post is to highlight the monitoring capabilities of Resource Governor.</p>
<blockquote>
<p>Throughout the examples we will be using a database called <code>DBA</code>. This is a database name I use to store all of my administration scripts, procedures, functions, lookup tables, etc. I create it on every instance I manage and it keeps a nice set of familiar tools close at hand, regardless of which instance I am working on. I highly suggest building your own DBA database.</p>
</blockquote>
<h1>The Basics</h1>
<p>Before we do anything with Resource Governor we have to make sure it's turned on:</p>
<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span><span class="w"> </span><span class="n">is_enabled</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">sys</span><span class="p">.</span><span class="n">resource_governor_configuration</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>If it's not enabled, it's simple to turn on:</p>
<div class="codehilite"><pre><span></span><code><span class="k">ALTER</span><span class="w"> </span><span class="n">RESOURCE</span><span class="w"> </span><span class="n">GOVERNOR</span><span class="w"> </span><span class="n">RECONFIGURE</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<h2>Resource Pools</h2>
<p>When thinking about Resource Governor, it can be helpful to think of it as a hierarchy of resource filters. At the very top, wide open, you have the whole of your SQL Server, all of the cores, all of the memory, all of the IO. Below that we have resource pools. At the pool level you can set the minimum and maximum CPU, memory, and IO, as well as the NUMA node affinity. The resources in the pool are shared among all session requests classified to a workload group within that pool.</p>
<p>Creating a pool is simple:</p>
<div class="codehilite"><pre><span></span><code><span class="k">CREATE</span><span class="w"> </span><span class="n">RESOURCE</span><span class="w"> </span><span class="n">POOL</span><span class="w"> </span><span class="n">Team01</span><span class="p">;</span><span class="w"> </span>
<span class="k">GO</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="n">RESOURCE</span><span class="w"> </span><span class="n">POOL</span><span class="w"> </span><span class="n">Team02</span><span class="p">;</span><span class="w"> </span>
<span class="k">GO</span><span class="w"> </span>
<span class="k">ALTER</span><span class="w"> </span><span class="n">RESOURCE</span><span class="w"> </span><span class="n">GOVERNOR</span><span class="w"> </span><span class="n">RECONFIGURE</span><span class="p">;</span><span class="w"> </span>
<span class="k">GO</span><span class="w"></span>
</code></pre></div>
<p>This creates two unrestricted pools, this will not limit the resources session requests within these pool can use.</p>
<p>When creating pools you have a few options for limiting session requests:</p>
<ul>
<li>Min/Max CPU limits (max CPU is applicable in times of CPU contention <em>only</em>)</li>
<li>Hard CPU cap (max CPU that is applicable at all times)</li>
<li>Min/Max query execution grant memory (be careful with the min setting on this one, as it reduces the max available memory for other pools)</li>
<li>Min/Max IOPS on a per volume basis</li>
<li>NUMA node affinity</li>
</ul>
<p>I could go into great detail on these settings, but I would really just be copying Microsofts own documentation word-for-word, as it is quite concise: <a href="https://msdn.microsoft.com/en-us/library/hh510189.aspx">MSDN</a></p>
<blockquote>
<p>One side-effect of classifying sessions into Resource Pools, is that each pool has it's own plan cache. While it may be a bit of overkill, you can use Resource Pools to eleviate bad parameter sniffing issues if you have two applications running similar queries with wildly different parameters.</p>
</blockquote>
<h2>Workload Groups</h2>
<p>Following our hierarchy down from the pool level, we get to the workload group. A workload group is restricted to using the resources defined by the pool it belongs to, and adds the additional ability to limit the maximum memory grant per request ( as a percentage of total memory in the pool ), MAXDOP, maximum concurrent requests allowed in the group, and more.</p>
<p>Again, creating a workload group is very straight forward, we'll create 4 groups, one for each application in our two resource pools:</p>
<div class="codehilite"><pre><span></span><code><span class="k">CREATE</span><span class="w"> </span><span class="n">WORKLOAD</span><span class="w"> </span><span class="k">GROUP</span><span class="w"> </span><span class="n">App01</span><span class="w"> </span>
<span class="k">USING</span><span class="w"> </span><span class="n">Team01</span><span class="p">;</span><span class="w"> </span>
<span class="k">GO</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="n">WORKLOAD</span><span class="w"> </span><span class="k">GROUP</span><span class="w"> </span><span class="n">App02</span><span class="w"> </span>
<span class="k">USING</span><span class="w"> </span><span class="n">Team01</span><span class="p">;</span><span class="w"> </span>
<span class="k">GO</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="n">WORKLOAD</span><span class="w"> </span><span class="k">GROUP</span><span class="w"> </span><span class="n">App03</span><span class="w"> </span>
<span class="k">USING</span><span class="w"> </span><span class="n">Team02</span><span class="p">;</span><span class="w"> </span>
<span class="k">GO</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="n">WORKLOAD</span><span class="w"> </span><span class="k">GROUP</span><span class="w"> </span><span class="n">App04</span><span class="w"> </span>
<span class="k">USING</span><span class="w"> </span><span class="n">Team02</span><span class="p">;</span><span class="w"> </span>
<span class="k">GO</span><span class="w"> </span>
<span class="k">ALTER</span><span class="w"> </span><span class="n">RESOURCE</span><span class="w"> </span><span class="n">GOVERNOR</span><span class="w"> </span><span class="n">RECONFIGURE</span><span class="p">;</span><span class="w"> </span>
<span class="k">GO</span><span class="w"></span>
</code></pre></div>
<p>Like the pools, these workload groups have no limits applied to them, so the groups are only used to classify the workload for reporting purposes.</p>
<h2>Classifier</h2>
<p>Once the pools and workload groups have been created, we have to write the function to classify our requests into these groups. When writing a classifier it's <em>very important</em> to remember that this function will be called for every single session, so we want to make it as lean and efficient as possible. The classifier only has one job: based on some condition, return the name of the workload group to assign the current session to.</p>
<p>Here is a simple example, this classifier will assign sessions from our four applications into their cooresponding workload groups:</p>
<div class="codehilite"><pre><span></span><code><span class="n">USE</span><span class="w"> </span><span class="n">master</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">FUNCTION</span><span class="w"> </span><span class="n">dbo</span><span class="p">.</span><span class="n">fnRGClassifier</span><span class="w"> </span><span class="p">()</span><span class="w"></span>
<span class="k">RETURNS</span><span class="w"> </span><span class="n">sysname</span><span class="w"></span>
<span class="k">WITH</span><span class="w"> </span><span class="n">SCHEMABINDING</span><span class="w"> </span>
<span class="k">AS</span><span class="w"> </span>
<span class="k">BEGIN</span><span class="w"></span>
<span class="w"> </span><span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">app_name</span><span class="w"> </span><span class="n">sysname</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">APP_NAME</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="k">RETURN</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="k">CASE</span><span class="w"></span>
<span class="w"> </span><span class="k">WHEN</span><span class="w"> </span><span class="o">@</span><span class="n">app_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">N</span><span class="s1">'Application 01'</span><span class="w"> </span><span class="k">THEN</span><span class="w"> </span><span class="n">N</span><span class="s1">'App01'</span><span class="w"></span>
<span class="w"> </span><span class="k">WHEN</span><span class="w"> </span><span class="o">@</span><span class="n">app_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">N</span><span class="s1">'Application 02'</span><span class="w"> </span><span class="k">THEN</span><span class="w"> </span><span class="n">N</span><span class="s1">'App02'</span><span class="w"></span>
<span class="w"> </span><span class="k">WHEN</span><span class="w"> </span><span class="o">@</span><span class="n">app_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">N</span><span class="s1">'Application 03'</span><span class="w"> </span><span class="k">THEN</span><span class="w"> </span><span class="n">N</span><span class="s1">'App03'</span><span class="w"></span>
<span class="w"> </span><span class="k">WHEN</span><span class="w"> </span><span class="o">@</span><span class="n">app_name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">N</span><span class="s1">'Application 04'</span><span class="w"> </span><span class="k">THEN</span><span class="w"> </span><span class="n">N</span><span class="s1">'App04'</span><span class="w"></span>
<span class="w"> </span><span class="k">ELSE</span><span class="w"> </span><span class="k">NULL</span><span class="w"></span>
<span class="w"> </span><span class="k">END</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="k">END</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">ALTER</span><span class="w"> </span><span class="n">RESOURCE</span><span class="w"> </span><span class="n">GOVERNOR</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="p">(</span><span class="n">CLASSIFIER_FUNCTION</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">dbo</span><span class="p">.</span><span class="n">fnRGClassifier</span><span class="p">)</span><span class="w"> </span>
<span class="k">ALTER</span><span class="w"> </span><span class="n">RESOURCE</span><span class="w"> </span><span class="n">GOVERNOR</span><span class="w"> </span><span class="n">RECONFIGURE</span><span class="w"> </span>
<span class="k">GO</span><span class="w"></span>
</code></pre></div>
<blockquote>
<p>Before you get too fancy, just remember, the classifier function has to be schema bound, so you will not be able to access objects outside of <code>master.dbo</code>. This elimates the posiblity of doing things like accessing lookup tables outside of master, or using in-memory tables or natively compiled functions.</p>
</blockquote>
<h2>Resource Governor DMVs</h2>
<p>There are a few DMVs available to help you view the configuration of Resource Governor as well as usage statistics on your pools and workload groups. If you want to verify the configuration of what we have done so far, and see whats statistics are available, run these simple statements:</p>
<div class="codehilite"><pre><span></span><code><span class="c1">-- Check the classifier is set properly and that Resource Governor is enabled</span>
<span class="k">SELECT</span><span class="w"> </span><span class="n">OBJECT_NAME</span><span class="p">(</span><span class="n">classifier_function_id</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">classifier</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">is_enabled</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">sys</span><span class="p">.</span><span class="n">resource_governor_configuration</span><span class="p">;</span><span class="w"></span>
<span class="c1">-- View our pools</span>
<span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">sys</span><span class="p">.</span><span class="n">dm_resource_governor_resource_pools</span><span class="p">;</span><span class="w"></span>
<span class="c1">-- View our workload groups</span>
<span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">sys</span><span class="p">.</span><span class="n">dm_resource_governor_workload_groups</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<blockquote>
<p>If you simply want to test to make sure your sessions are being properly classified, <code>sys.dm_exec_sessions</code> has a column called <code>group_id</code> that you can use to join on <code>sys.dm_resource_governor_workload_groups</code> to see which group each session has been classified to.</p>
</blockquote>
<p>From looking at these DMVs it's pretty obvious how much data is available on your resource pools and workload groups. For those of you not following along, these DMVs allow you to view the following (and more):</p>
<ul>
<li>Current/total request count</li>
<li>Current/total requests queued (if these sessions would violate your limits)</li>
<li>Total lock wait time and count</li>
<li>Total CPU usage in milliseconds</li>
<li>Total resource limit violations</li>
<li>Total reads/writes</li>
<li>Total read/write IO stalls</li>
</ul>
<p>Depending on the statistic you want, you can get them at the pool or group level, or in some cases both. You can read more about the available DMVs here: <a href="https://msdn.microsoft.com/en-us/library/bb934218.aspx">MSDN</a></p>
<blockquote>
<p>In addition to all of this great data, if you impose IO limits at the resource pool level you will also have access to <code>sys.dm_resource_governor_resource_pool_volumes</code>. This DMV will allow you to dig deep into the IO usage of your pools on a per volume basis. If you are currently running a IO-bound workload, I would highly suggest taking a look at what Resource Governor can offer here.</p>
</blockquote>
<h1>Collecting Data</h1>
<p>Now that we know <em>which</em> statistics we can gather, we need to actually start gathering them. While the DMVs for Resource Governor are great, they will only give you an aggregate of the usage information since the last time the statistics were reset, or the last time services were restarted. </p>
<p>In most cases it makes sense to store your data in a separate table so you can calculate differentials between two time periods. For our example we are only going to be interested in request counts and CPU usage. For this, we will create the following table:</p>
<div class="codehilite"><pre><span></span><code><span class="n">USE</span><span class="w"> </span><span class="n">DBA</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">dbo</span><span class="p">.</span><span class="n">ResourceGovernorUsageData</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">CollectionTimeUTC</span><span class="w"> </span><span class="n">DATETIME</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">CollectionTimeOffset</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">ResourcePool</span><span class="w"> </span><span class="n">SYSNAME</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">WorkloadGroup</span><span class="w"> </span><span class="n">SYSNAME</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">RequestCountTotal</span><span class="w"> </span><span class="nb">BIGINT</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">RequestCountDelta</span><span class="w"> </span><span class="nb">BIGINT</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">CPUUsageMSTotal</span><span class="w"> </span><span class="nb">BIGINT</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">CPUUsageMSDelta</span><span class="w"> </span><span class="nb">BIGINT</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="k">CONSTRAINT</span><span class="w"> </span><span class="n">PK_ResourceGovernorUsageData</span><span class="w"> </span><span class="k">PRIMARY</span><span class="w"> </span><span class="k">KEY</span><span class="w"> </span><span class="n">CLUSTERED</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">CollectionTimeUTC</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">ResourcePool</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">WorkloadGroup</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="p">)</span><span class="w"></span>
</code></pre></div>
<p>With the table created, we will also create a stored procedure to write data to the table, eventually calling the procedure via a scheduled job. The general idea behind the procedure is that it will grab the current statistics, and then read the statistics currently being stored in the table. It will then write the new statistics it gathered, as well as the delta between this run of the procedure and the last. Technically you could just store the aggregated value as it appears in the DMVs, but the query needed to process that data would end up being costly.</p>
<p>When writing a procedure like this you have to account for two cases:</p>
<ul>
<li>The DMV statistics get reset: This would result in a negative delta, as the latest total would be less than the previous</li>
<li>The first time you load the data: If you are starting with nothing, there is no delta to store, in which case you have to decide if you want to store a delta of <code>0</code> or <code>NULL</code></li>
</ul>
<p>Here is an example of a procedure we could use to store our data:</p>
<div class="codehilite"><pre><span></span><code><span class="n">USE</span><span class="w"> </span><span class="n">DBA</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">PROCEDURE</span><span class="w"> </span><span class="n">dbo</span><span class="p">.</span><span class="n">Collect_ResourceGovernorStats</span><span class="w"></span>
<span class="k">AS</span><span class="w"></span>
<span class="p">;</span><span class="k">WITH</span><span class="w"> </span><span class="n">RGCTE</span><span class="w"> </span><span class="k">AS</span><span class="w"></span>
<span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="n">GETUTCDATE</span><span class="p">()</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">now_utc</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">DATEDIFF</span><span class="p">(</span><span class="n">HOUR</span><span class="p">,</span><span class="n">GETUTCDATE</span><span class="p">(),</span><span class="n">GETDATE</span><span class="p">())</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">now_offset</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">rgp</span><span class="p">.</span><span class="n">name</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">pool_name</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">rgg</span><span class="p">.</span><span class="n">name</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">group_name</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">rgg</span><span class="p">.</span><span class="n">total_request_count</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">rgg</span><span class="p">.</span><span class="n">total_cpu_usage_ms</span><span class="w"></span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">sys</span><span class="p">.</span><span class="n">resource_governor_resource_pools</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">rgp</span><span class="w"></span>
<span class="w"> </span><span class="k">INNER</span><span class="w"> </span><span class="k">JOIN</span><span class="w"> </span><span class="n">sys</span><span class="p">.</span><span class="n">dm_resource_governor_workload_groups</span><span class="w"> </span><span class="k">As</span><span class="w"> </span><span class="n">rgg</span><span class="w"></span>
<span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="n">rgp</span><span class="p">.</span><span class="n">pool_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">rgg</span><span class="p">.</span><span class="n">pool_id</span><span class="w"></span>
<span class="p">)</span><span class="w"></span>
<span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">ResourceGovernorUsageData</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">CollectionTimeUTC</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">CollectionTimeOffset</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">ResourcePool</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">WorkloadGroup</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">RequestCountTotal</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">RequestCountDelta</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">CPUUsageMSTotal</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">CPUUsageMSDelta</span><span class="w"></span>
<span class="p">)</span><span class="w"> </span>
<span class="k">SELECT</span><span class="w"> </span><span class="n">RGCTE</span><span class="p">.</span><span class="n">now_utc</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">RGCTE</span><span class="p">.</span><span class="n">now_offset</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">RGCTE</span><span class="p">.</span><span class="n">group_name</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">RGCTE</span><span class="p">.</span><span class="n">pool_name</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">RGCTE</span><span class="p">.</span><span class="n">total_request_count</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="k">CASE</span><span class="w"></span>
<span class="w"> </span><span class="k">WHEN</span><span class="w"> </span><span class="n">RGCTE</span><span class="p">.</span><span class="n">total_request_count</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="k">ISNULL</span><span class="p">(</span><span class="n">LastRun</span><span class="p">.</span><span class="n">RequestCountTotal</span><span class="p">,</span><span class="mi">0</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">THEN</span><span class="w"> </span><span class="n">RGCTE</span><span class="p">.</span><span class="n">total_request_count</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">LastRun</span><span class="p">.</span><span class="n">RequestCountTotal</span><span class="w"> </span><span class="k">ELSE</span><span class="w"> </span><span class="mi">0</span><span class="w"></span>
<span class="w"> </span><span class="k">END</span><span class="w"></span>
<span class="w"> </span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="n">RGCTE</span><span class="p">.</span><span class="n">total_cpu_usage_ms</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="k">CASE</span><span class="w"></span>
<span class="w"> </span><span class="k">WHEN</span><span class="w"> </span><span class="n">RGCTE</span><span class="p">.</span><span class="n">total_cpu_usage_ms</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="k">ISNULL</span><span class="p">(</span><span class="n">LastRun</span><span class="p">.</span><span class="n">CPUUsageMSTotal</span><span class="p">,</span><span class="mi">0</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="k">THEN</span><span class="w"> </span><span class="n">RGCTE</span><span class="p">.</span><span class="n">total_cpu_usage_ms</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="n">LastRun</span><span class="p">.</span><span class="n">CPUUsageMSTotal</span><span class="w"> </span><span class="k">ELSE</span><span class="w"> </span><span class="mi">0</span><span class="w"></span>
<span class="w"> </span><span class="k">END</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">RGCTE</span><span class="w"></span>
<span class="w"> </span><span class="k">OUTER</span><span class="w"> </span><span class="n">APPLY</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="n">TOP</span><span class="w"> </span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">RequestCountTotal</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">CPUUsageMSTotal</span><span class="w"></span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">ResourceGovernorUsageData</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">RGD</span><span class="w"></span>
<span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">RGD</span><span class="p">.</span><span class="n">ResourcePool</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">RGCTE</span><span class="p">.</span><span class="n">pool_name</span><span class="w"></span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">RGD</span><span class="p">.</span><span class="n">WorkloadGroup</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">RGCTE</span><span class="p">.</span><span class="n">group_name</span><span class="w"></span>
<span class="w"> </span><span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"></span>
<span class="w"> </span><span class="n">CollectionTimeUTC</span><span class="w"> </span><span class="k">DESC</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">LastRun</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>This procedure will take a snapshot of the current resource governor stats (via the CTE), and then subtract the most recent stats from our table and store the new total and the delta. We are also handling our error cases, making sure to throw out our delta if the current aggregated total is not higher than the previous, and also throwing out our data if there is no data in the table to calculate a delta from.</p>
<p>Now all you have to do is schedule this procedure to run on a regular basis (I chose to run this every 5 minutes via SQL Agent) and you'll have a nice high level view of resource consumption on your instance.</p>
<h1>Where to Go from Here</h1>
<p>For me, this is the really fun part. There is a lot you can do with this data. Even if you just leave the data on the instance and export it to excel for analysis after new code releases it can be a very valuable tool. </p>
<p>In my environment I am using a custom PowerShell module to simultaniously write my Resource Governor data to SQL Server as well as Elasticsearch. Once the data is in Elasticsearch I can run aggregations on it, analyse it with Python, and create resource usage dashboards for the various team managers.</p>
<p>Look for future posts where I talk about my custom module and provide visual examples of what can be done in Kibana once you get this data into Elasticsearch.</p>getting started with in-memory oltp: reducing tempdb contention2017-06-21tempdb-contentionReducing TempDB contention with memory optimized table variables.<p>Everyone loves the temporary table. temporary tables can be very useful when trying to execute complex operations, or get around optimizer quirks. But what happens if you love them a little too much? In this post we'll go through some common tempdb latch contention scenarios you might only see under extremely heavy load, and how you can use memory-optimized table variables to remove or reduce this contention.</p>
<blockquote>
<p>In-Memory OLTP is a new technology that requires some research before implementation, but hopefully this post gets you interested enough to give it a try.</p>
</blockquote>
<h1>Tempdb Latch Contention</h1>
<p>In my environment we have a highly concurrent workload and make heavy use of tempdb. In the past we have run into the common tempdb contention issues most people run into, PFS/SGAM contention. This was easy to fix by adding more tempdb files, but after a change in workload patterns we started seeing contention in a new place: the base system tables in tempdb. </p>
<blockquote>
<p>For more information on troubleshooting PFS/SGAM contention in tempdb, please refer to this post by Jonathan Kehayias: <a href="https://www.simple-talk.com/sql/database-administration/optimizing-tempdb-configuration-with-sql-server-2012-extended-events/">Optimizing tempdb configuration with SQL Server 2012 Extended Events</a></p>
</blockquote>
<h1>Understanding Whats Happening</h1>
<p>Before you can understand why this contention occurs, it is important to get a basic understanding of what happens in SQL Server when you create a temporary table. Whenever you create a temporary table, entries are made in various system tables in tempdb just as if you were creating a normal table in a user database, and in many cases the temporary table definition is cached.</p>
<p>Cached tables are stored in <code>tempdb.sys.tables</code> with a hex value in place of the name they were given when they were created. When you create a table that is already in cache, a simple rename operation occurs instead of an insert into the system tables. This will be important to understand later, and this is about all we are going to cover in this post regarding temporary table caching.</p>
<blockquote>
<p>Temporary table caching is a complex topic that is outside the scope of this post, but is covered beautifully by Paul White in his post <a href="http://sqlblog.com/blogs/paul_white/archive/2012/08/17/temporary-object-caching-explained.aspx">Temporary Table Caching Explained</a></p>
</blockquote>
<p>If you create a temporary table then query <code>tempdb.sys.tables</code> you will see a list of cached objects (named with a hex value) as well as an entry for the temporary table you just created:</p>
<div class="codehilite"><pre><span></span><code><span class="c1">--== Create a table</span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="o">#</span><span class="n">Test</span><span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="n">ID</span><span class="w"> </span><span class="nb">INT</span><span class="w"> </span><span class="p">);</span><span class="w"></span>
<span class="c1">--== See all the tables</span>
<span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">tempdb</span><span class="p">.</span><span class="n">sys</span><span class="p">.</span><span class="n">tables</span><span class="p">;</span><span class="w"></span>
<span class="c1">--== See just our table</span>
<span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">tempdb</span><span class="p">.</span><span class="n">sys</span><span class="p">.</span><span class="n">tables</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"> </span><span class="n">name</span><span class="w"> </span><span class="k">LIKE</span><span class="w"> </span><span class="s1">'#Test%'</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>If you also query <code>tempdb.sys.columns</code> for the associated <code>object_id</code>, you will see what you would expect in a standard user database, a list of columns for your temporary table. Where we get into trouble is when we are creating these temporary tables at an extremely high rate. A quick query will show you why:</p>
<div class="codehilite"><pre><span></span><code><span class="n">USE</span><span class="w"> </span><span class="n">tempdb</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">EXEC</span><span class="w"> </span><span class="n">sp_help</span><span class="w"> </span><span class="s1">'sys.sysschobjs'</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
</code></pre></div>
<blockquote>
<p><em>sp_help</em> is quite possibly the most under-appreciated system stored procedure on your instances. I highly recommend playing around with it to see what you can find. </p>
</blockquote>
<p>Running the above query will give you valuable details about the system base table <code>sys.sysschobjs</code>, the base table for <code>sys.tables</code>. An interesting thing to note is that there are 4 indexes on this table, one of which is a non-clustered index that leads with the <code>name</code> column, and another that leads with a <code>TINYINT</code> column named <code>nsclass</code>. In situations where you have potentially hundreds of client requests attempting to create a temporary table it is very possible to start seeing contention on this table. Below I will show you how to reproduce this contention, and how to investigate any live contention you might see on your own instances.</p>
<h1>Creating Latch Contention</h1>
<p>To recreate this sort of contention we are going to use the OSTRESS command line utility (Get it here: <a href="https://support.microsoft.com/en-us/kb/944837">https://support.microsoft.com/en-us/kb/944837</a>). <code>ostress</code> is a simple command line utility you can use to execute SQL queries using multiple threads. This will need to be installed before continuing on with any of the demos below.</p>
<p>To prepare for our contention test we need to create a test database and define a stored procedure that simply creates a temporary table. </p>
<div class="codehilite"><pre><span></span><code><span class="n">USE</span><span class="w"> </span><span class="n">master</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">DATABASE</span><span class="w"> </span><span class="n">ContentionTest</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="n">USE</span><span class="w"> </span><span class="n">ContentionTest</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">PROCEDURE</span><span class="w"> </span><span class="n">dbo</span><span class="p">.</span><span class="n">Test</span><span class="w"></span>
<span class="k">AS</span><span class="w"> </span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="o">#</span><span class="n">Test</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">Id</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">Col1</span><span class="w"> </span><span class="n">NVARCHAR</span><span class="p">(</span><span class="mi">128</span><span class="p">)</span><span class="w"></span>
<span class="p">);</span><span class="w"></span>
<span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="o">#</span><span class="n">Test</span><span class="w"></span>
<span class="k">SELECT</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="s1">'Test'</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>With our procedure defined we can fire up the <code>RML Cmd Prompt</code> ( now found in your start menu ) and run ostress. Once at the RML prompt, type the following command ( in our example we are running this against a local instance ):</p>
<div class="codehilite"><pre><span></span><code>ostress -Q<span class="s2">"EXEC ContentionTest.dbo.Test;"</span> -n500 -r500 -S<span class="s2">"localhost"</span>
</code></pre></div>
<blockquote>
<p>WARNING: Do not run this test on a production instance. Depending on the configuration of the instance you could quickly exhaust the worker thread pool, making the instance unresponsive.</p>
</blockquote>
<p>This will fire off 500 threads (-n), each executing our stored procedure 500 times in succession (-r). This test will take a minute to execute, even on a fairly capable machine. While this is happening, lets see if we have any contention. If you need to extend the duration of this command, increasing the <code>-r</code> value should do the trick.</p>
<p>Open SSMS and connect to the instance in question and run the following:</p>
<div class="codehilite"><pre><span></span><code><span class="n">USE</span><span class="w"> </span><span class="n">master</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">SELECT</span><span class="w"> </span><span class="n">es</span><span class="p">.</span><span class="n">session_id</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">es</span><span class="p">.</span><span class="n">login_time</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">er</span><span class="p">.</span><span class="n">wait_type</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">er</span><span class="p">.</span><span class="n">wait_resource</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">er</span><span class="p">.</span><span class="n">command</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">DB_NAME</span><span class="p">(</span><span class="n">er</span><span class="p">.</span><span class="n">database_id</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">dbname</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">sys</span><span class="p">.</span><span class="n">dm_exec_requests</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">er</span><span class="w"></span>
<span class="w"> </span><span class="k">INNER</span><span class="w"> </span><span class="k">JOIN</span><span class="w"> </span><span class="n">sys</span><span class="p">.</span><span class="n">dm_exec_sessions</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">es</span><span class="w"></span>
<span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="n">er</span><span class="p">.</span><span class="n">session_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">es</span><span class="p">.</span><span class="n">session_id</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"> </span><span class="n">es</span><span class="p">.</span><span class="n">is_user_process</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>If the test is running properly you will likely see a lot of sessions with a wait type of <code>PAGELATCH_EX</code>, waiting on some random database page. In my test for example, waits primarily occurred on a wait resource identified as: <code>2:1:14469</code></p>
<p>This can be decoded to mean that we have <code>PAGELATCH_EX</code> waits on page <code>14469</code> of file <code>1</code> in database <code>2</code> (tempdb). So what's on this page? Let's take a look:</p>
<div class="codehilite"><pre><span></span><code><span class="n">DBCC</span><span class="w"> </span><span class="n">PAGE</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">14469</span><span class="p">,</span><span class="mi">3</span><span class="p">)</span><span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="n">TABLERESULTS</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<blockquote>
<p>Notice we just took the wait resource from the results of the query above, traded out our colons for commas, and added a 3</p>
</blockquote>
<p>What you'll typically get back are two result sets depending on the type of page you are looking at. The first result set contains, among other information, the page header. The second is a dump of the actual data on the page. For this example we are interested in the first result set, we are looking for two rows with the following field names: <code>Metadata: ObjectId</code> and <code>Metadata: IndexId</code>. In my case I am seeing 34 and 2 respectively.</p>
<p>Now that we know the object id of the table we are seeing contention on we can use <code>sys.objects</code> to look up the table name, and use the index id to see which index this contention is on:</p>
<div class="codehilite"><pre><span></span><code><span class="n">USE</span><span class="w"> </span><span class="n">tempdb</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">SELECT</span><span class="w"> </span><span class="n">name</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">sys</span><span class="p">.</span><span class="n">objects</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"> </span><span class="n">object_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">34</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>If all worked, you should now see that we have contention on the <code>sysschobjs</code> table. Earlier we discussed using <code>sp_help</code> to get index details on system tables, if we do that now and look at index 2, we will see the lead column is <code>nsclass</code> which is a tinyint field. Using a tinyint as a lead column is typically a terrible idea since there is little selectivity on such a narrow field, and this is no exception. </p>
<p>This isn't the only case of contention you might see with system objects related to temporary tables. We ran into a few different contention scenarios with tempdb:</p>
<ul>
<li>Contention on <code>sysschobjs</code> again, but on index 3. This index leads with the name of the temporary table and is fairly narrow so you can fit a lot of records on a single index page. Because of this, if you are running lots of concurrent procedures that create temporary tables with the same or similar names, it creates a hot spot on a single page, leading to more contention.</li>
<li>Temporary table auto-stats. Statistics objects for all tables (including temporary tables) are stored in the <code>sys.sysobjvalues</code> table. If you get enough auto-stats generations on temporary tables you can see contention here.</li>
</ul>
<p>So now you know where your contention is, what can you do about it?</p>
<h1>Reducing Contention with In-Memory OLTP</h1>
<p>SQL Server In-Memory OLTP (or Hekaton) is a new database engine that runs along side of the classic database engine you are used to using. You can query in-memory/memory-optimized objects just like their disk-based counterparts. The major difference is that memory-optimized objects operate using truely optimistic concurrency, allowing you extremely fast, lock-free access to your data. There is a lot to understand when you start out using memory-optimized objects, but it's fairly easy to get your feet wet with memory-optimized table variables. </p>
<h2>Memory-Optimized Table Variables</h2>
<p>Memory-optimized table variables are just standard table variables that use a user-defined memory-optimized table type. Creating the table type is easy, but in order to do so you will need a memory-optimized filegroup and container in your database. </p>
<p><strong>Create the filegroup:</strong></p>
<div class="codehilite"><pre><span></span><code><span class="k">ALTER</span><span class="w"> </span><span class="k">DATABASE</span><span class="w"> </span><span class="n">ContentionTest</span><span class="w"></span>
<span class="k">ADD</span><span class="w"> </span><span class="n">FILEGROUP</span><span class="w"> </span><span class="n">imoltp</span><span class="w"> </span><span class="k">CONTAINS</span><span class="w"> </span><span class="n">MEMORY_OPTIMIZED_DATA</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p><strong>Create the container:</strong></p>
<div class="codehilite"><pre><span></span><code><span class="k">ALTER</span><span class="w"> </span><span class="k">DATABASE</span><span class="w"> </span><span class="n">ContentionTest</span><span class="w"> </span><span class="k">ADD</span><span class="w"> </span><span class="n">FILE</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="n">name</span><span class="o">=</span><span class="s1">'imoltp01'</span><span class="p">,</span><span class="w"> </span><span class="n">filename</span><span class="o">=</span><span class="s1">'c:\data\imoltp'</span><span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="k">TO</span><span class="w"> </span><span class="n">FILEGROUP</span><span class="w"> </span><span class="n">imoltp</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p><strong>Create a schema and table type:</strong></p>
<div class="codehilite"><pre><span></span><code><span class="n">USE</span><span class="w"> </span><span class="n">ContentionTest</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">SCHEMA</span><span class="w"> </span><span class="n">MemoryOptimized</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">TYPE</span><span class="w"> </span><span class="n">MemoryOptimized</span><span class="p">.</span><span class="n">IdTable</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span>
<span class="p">(</span><span class="w"> </span>
<span class="w"> </span><span class="n">Id</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">Col1</span><span class="w"> </span><span class="n">NVARCHAR</span><span class="p">(</span><span class="mi">128</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="k">PRIMARY</span><span class="w"> </span><span class="k">KEY</span><span class="w"> </span><span class="n">NONCLUSTERED</span><span class="w"> </span><span class="p">(</span><span class="n">Id</span><span class="p">)</span><span class="w"> </span>
<span class="p">)</span><span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span><span class="n">MEMORY_OPTIMIZED</span><span class="o">=</span><span class="k">ON</span><span class="p">);</span><span class="w"> </span><span class="c1">-- <== The magic happens here.</span>
<span class="k">GO</span><span class="w"></span>
</code></pre></div>
<blockquote>
<p>I created a schema called 'MemoryOptimized' to create my table type under. How you organize your objects is your business, but as you start using memory-optimized objects, I highly recommend placing these objects in their own schema just for the sake of clarity.</p>
</blockquote>
<p>Now that we have our new table type created, we can use it in a procedure:</p>
<div class="codehilite"><pre><span></span><code><span class="n">USE</span><span class="w"> </span><span class="n">ContentionTest</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">PROCEDURE</span><span class="w"> </span><span class="n">dbo</span><span class="p">.</span><span class="n">Test2</span><span class="w"></span>
<span class="k">AS</span><span class="w"> </span>
<span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">Test</span><span class="w"> </span><span class="n">MemoryOptimized</span><span class="p">.</span><span class="n">IdTable</span><span class="p">;</span><span class="w"></span>
<span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="o">@</span><span class="n">Test</span><span class="w"></span>
<span class="k">SELECT</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="s1">'Test'</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>And that's it! Nothing too exciting around the implementation of memory-optimized table variables, but the results are pretty amazing. Let's run ostress again, using our new procedure, and see what happens to our instance:</p>
<div class="codehilite"><pre><span></span><code>ostress -Q<span class="s2">"EXEC ContentionTest.dbo.Test2;"</span> -n500 -r500 -S<span class="s2">"localhost"</span>
</code></pre></div>
<p>Now lets run our query to see what's happening on the instance:</p>
<div class="codehilite"><pre><span></span><code><span class="n">USE</span><span class="w"> </span><span class="n">master</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">SELECT</span><span class="w"> </span><span class="n">es</span><span class="p">.</span><span class="n">session_id</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">es</span><span class="p">.</span><span class="n">login_time</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">er</span><span class="p">.</span><span class="n">wait_type</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">er</span><span class="p">.</span><span class="n">wait_resource</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">er</span><span class="p">.</span><span class="n">command</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">DB_NAME</span><span class="p">(</span><span class="n">er</span><span class="p">.</span><span class="n">database_id</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">dbname</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">sys</span><span class="p">.</span><span class="n">dm_exec_requests</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">er</span><span class="w"></span>
<span class="w"> </span><span class="k">INNER</span><span class="w"> </span><span class="k">JOIN</span><span class="w"> </span><span class="n">sys</span><span class="p">.</span><span class="n">dm_exec_sessions</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">es</span><span class="w"></span>
<span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="n">er</span><span class="p">.</span><span class="n">session_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">es</span><span class="p">.</span><span class="n">session_id</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"> </span><span class="n">es</span><span class="p">.</span><span class="n">is_user_process</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>You should now notice that it's hard to even catch an active session. All pagelatch contention is gone, you'll even notice that your ostress session finishes MUCH faster. So why did this work? It's simple: we moved our temporary table operation out of tempdb and into the user database, and on top of that we moved it into a memory-optimized object.</p>
<h1>Final Thoughts</h1>
<p>The results of this simple change can be dramatic, but before you get carried away and try to convert all of your temporary tables over to memory optimized table types there are a few things to keep in mind about in-memory OLTP, and tempdb contention:</p>
<ul>
<li>
<p>If you are on SQL Server 2014, prior to SP1 CU4, you can run into issues where the XTP checkpoint thread dies and prevents log truncation on the database it belongs to. This can result in your tlogs growing out of control until you take the database offline.</p>
</li>
<li>
<p>You CANNOT remove a memory-optimized filegroup from a database without dropping the entire database.</p>
</li>
<li>
<p>It is highly recommended that you use resource governor to bind any databases that use memory-optimized objects to their own resource pool with memory limits configured. This can prevent a sudden growth of in-memory data from starving the <code>default</code> pool of memory.</p>
</li>
<li>
<p>While implementing memory-optimized table variables is fairly straight-forward, things can get more complicated when you start using memory-optimized tables. The biggest issue you'll likely see is that you cannot use cross-database transactions if a memory-optimized table is involved. This was one of the main driving factors when we decided to go with memory-optimized table types over memory-optimized tables.</p>
</li>
<li>
<p>If you are on a version of SQL Server 2016 prior to SP1 CU2, or prior to 2016 RTM CU6, the demos above should work as described. If you have applied those updates the results will be less dramatic. Working with the SQLCAT team on this issue, we discovered a code path in SQL Server 2016 that introduced additional latches when dealing with cached temporary tables. This lead to an almost 100% increase in pagelatch wait types when using standard temporary tables (<a href="https://support.microsoft.com/en-us/help/4013999">KB4013999</a>).</p>
</li>
</ul>
<p>The above list can be a little scary, but it's all worth it in the end. In-Memory OLTP is here to stay, and the performance gains you can see from it can be very impressive. Make sure you check out the resources below for more details on everything we discussed.</p>
<h1>Resources</h1>
<ul>
<li><a href="https://www.simple-talk.com/sql/database-administration/optimizing-tempdb-configuration-with-sql-server-2012-extended-events/">Optimizing tempdb configuration with SQL Server 2012 Extended Events - Jonathan Kehayias</a></li>
<li><a href="http://sqlblog.com/blogs/paul_white/archive/2012/08/17/temporary-object-caching-explained.aspx">Temporary Table Caching Explained - Paul White</a></li>
<li><a href="https://msdn.microsoft.com/en-us/library/dn465873.aspx">Bind a Database with Memory-Optimized Tables to a Resource Pool - MSDN</a></li>
<li><a href="https://msdn.microsoft.com/en-us/library/dn282389.aspx">Estimate Memory Requirements for Memory-Optimized Tables - MSDN</a></li>
<li><a href="https://msdn.microsoft.com/en-us/library/dn133186.aspx">In-Memory OLTP (In-Memory Optimization)</a></li>
</ul>managing the sql server error log2017-07-13managing-errorlogWe typically think of error logs as somewhere to go to find issues, but what if your error logs ARE the issue?<p>We typically think of error logs as somewhere to go to find issues, but what if your error logs ARE the issue? Like most anything else in SQL Server, if you neglect your error logs you can run into trouble. Even on a low-traffic SQL Server instance, a bad piece of code, or a hardware issue, could easily fill your error logs, and with the introduction of Hekaton in SQL Server 2014, the SQL Server error log started getting a lot more data pumped into it than you might have been used to before. What this means for the DBA is that you can quickly start filling your main system drive (if your SQL install and error logs are in the default location) with massive error logs. So what questions should you be answering about error logs to make sure you don't run into problems?</p>
<ol>
<li>Where are SQL Error Logs stored?</li>
<li>How long are error logs kept?</li>
<li>How much space do I need for error logs?</li>
</ol>
<h1>Where are SQL Error Logs Stored?</h1>
<p>The quickest way to answer this question is to run a simple query:</p>
<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span><span class="w"> </span><span class="n">SERVERPROPERTY</span><span class="p">(</span><span class="s1">'ErrorLogFileName'</span><span class="p">)</span><span class="w"></span>
</code></pre></div>
<p>This will return the full path of your current error log. By default this will be located under <code>C:\Program Files\Microsoft SQL Server\</code>.</p>
<p>The location of your logs can be easily changed by adding a startup parameter to the SQL Server service in configuration manager. Simply adding a <code>-e</code> parameter, followed by the desired error log path, will change the path error logs are written to on next service restart. If for example you had a dedicated drive and directory, <code>E:\Logs</code>, that you wanted to write your error logs to, you would add the following startup parameter: <code>-eE:\LOGS</code>.</p>
<blockquote>
<p>When adding new startup parameters, it is important to remember that parameters are semi-colon delimited.</p>
</blockquote>
<h1>How Long are Error Logs Kept?</h1>
<p>This question can depend on what, if any, maintenance is being done on your error logs. If you cycle error logs nightly (via a SQL Agent job that executes <code>EXEC sp_cycle_errorlog</code>), you can check the number of SQL error logs that will be kept before re-using the files to determine roughly how many days worth of logs would be kept.</p>
<p>To find the number of error logs that are kept, navigate to the 'SQL Server Logs' node in the object explorer in Management Studio, right click, and select 'Configure'.</p>
<p><img alt="Statistic Histogram" src="/img/ConfigureLogFiles01.png" /></p>
<p>From here you can adjust the number of error log files to retain.</p>
<p><img alt="Statistic Histogram" src="/img/ConfigureLogFiles02.png" /></p>
<p>If you don't currently do such maintenance it can be hard to say. A new error log is created every time the SQL Server service restarts, or the <code>sp_cycle_errorlog</code> procedure is executed. If your instances don't normally restart there is no real limit to how large the file can get, and therefore, how far back the error log history goes.</p>
<blockquote>
<p>If you plan on cycling your logs nightly, and configuring a max number of logs, you should keep in mind that there is a hard limit of 99 files. If you want to keep 90 days worth of logs so you set the number of files to 90, but have 10 restarts within those 90 days, some of your log files will be lost.</p>
</blockquote>
<h1>How Much Space do I Need for Error Logs?</h1>
<p>As stated above, there isn't a practical limit to the size of the error log so the space required for error logs depends on the number of files you keep, how often the error log is cycled, and (if you are using SQL Server 2012+) whether or not a maximum error log file size has been set.</p>
<p>You can check to see if a limit has been set by checking:</p>
<p>For SQL Server 2012:</p>
<div class="codehilite"><pre><span></span><code><span class="n">USE</span><span class="w"> </span><span class="p">[</span><span class="n">master</span><span class="p">];</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">EXEC</span><span class="w"> </span><span class="n">xp_instance_regread</span><span class="w"></span>
<span class="w"> </span><span class="n">N</span><span class="s1">'HKEY_LOCAL_MACHINE'</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">N</span><span class="s1">'Software\Microsoft\MSSQLServer\MSSQLServer'</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">N</span><span class="s1">'ErrorLogSizeInKb'</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
</code></pre></div>
<p>For SQL Server 2014+</p>
<div class="codehilite"><pre><span></span><code><span class="n">USE</span><span class="w"> </span><span class="p">[</span><span class="n">master</span><span class="p">];</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">EXEC</span><span class="w"> </span><span class="n">xp_instance_regread</span><span class="w"></span>
<span class="w"> </span><span class="n">N</span><span class="s1">'HKEY_LOCAL_MACHINE'</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">N</span><span class="s1">'SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQLServer'</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">N</span><span class="s1">'ErrorLogSizeInKb'</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
</code></pre></div>
<blockquote>
<p>This key could change, so you may have to open up the registry editor (<code>regedit</code>) to search for this key if the above commands don't work for you.</p>
</blockquote>
<p>Changing this value is just as easy. Instead of executing an <code>xp_instance_regread</code>, we execute <code>xp_instance_regwrite</code>:</p>
<div class="codehilite"><pre><span></span><code><span class="n">USE</span><span class="w"> </span><span class="p">[</span><span class="n">master</span><span class="p">];</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">EXEC</span><span class="w"> </span><span class="n">xp_instance_regwrite</span><span class="w"></span>
<span class="w"> </span><span class="n">N</span><span class="s1">'HKEY_LOCAL_MACHINE'</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">N</span><span class="s1">'SOFTWARE\Microsoft\Microsoft SQL Server\MSSQL12.MSSQLSERVER\MSSQLServer'</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">N</span><span class="s1">'ErrorLogSizeInKb'</span><span class="w"></span>
<span class="w"> </span><span class="n">REG_DWORD</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="mi">1073741824</span><span class="p">;</span><span class="w"> </span><span class="c1">-- 1GB</span>
<span class="k">GO</span><span class="w"></span>
</code></pre></div>
<blockquote>
<p>These commands all assume you are using the default instance. To try this with different versions of SQL Server and named instances, you would have to replace the <code>MSSQL12</code> above with the version you are currently running (<code>MSSQL13</code> for SQL Server 2016), and <code>.MSSQLSERVER</code> with the name of named instance.</p>
</blockquote>
<h1>Final Thoughts</h1>
<p>Error log management can be an important part of managing any SQL Server. Left to their own devices, error logs can become enormous, making it harder for you to open them for troubleshooting purposes, and potentially leading to a full system drive. Cycling the log on a daily or weekly basis, and setting a sane limit on the number of files can go a long way towards keeping these files under control.</p>handling sql agent job failures2017-08-30job-failuresQuiet down your noisy job failure alerts and gain new functionality.<p>Hopefully you don't experience a lot of job failures. But when multiple high-frequency jobs start failing, your inbox can get overloaded. Besides being annoying, an inbox full of job failure emails can make it hard to sift out the signal from the noise. In this post we'll walk through a system that I have used to monitor job failures on 100's of instances running 150+ jobs a piece, many of which run every 5 minutes.</p>
<h1>Overview</h1>
<p>At it's core, the system has three components: a stored procedure, a SQL Agent job, and a table. The table stores job failure details. The agent job executes a procedure to copy job failure details to the table, and cleans up the job history table in MSDB.</p>
<h2>The Table</h2>
<p>The table is simple, and holds the information you would expect:
:::sql
USE [DBA]
GO</p>
<div class="codehilite"><pre><span></span><code><span class="k">CREATE</span><span class="w"> </span><span class="nc">TABLE</span><span class="w"> </span><span class="o">[</span><span class="n">Maintenance</span><span class="o">]</span><span class="p">.</span><span class="o">[</span><span class="n">JobFailureArchive</span><span class="o">]</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">JobFailureArchiveID</span><span class="w"> </span><span class="nc">INT</span><span class="w"> </span><span class="k">IDENTITY</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="ow">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">JobID</span><span class="w"> </span><span class="nc">UNIQUEIDENTIFIER</span><span class="w"> </span><span class="ow">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">JobName</span><span class="w"> </span><span class="n">SYSNAME</span><span class="w"> </span><span class="ow">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">RunDate</span><span class="w"> </span><span class="nc">DATETIME</span><span class="w"> </span><span class="ow">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">StepID</span><span class="w"> </span><span class="nc">INT</span><span class="w"> </span><span class="ow">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">OutcomeText</span><span class="w"> </span><span class="nc">NVARCHAR</span><span class="p">(</span><span class="mi">4000</span><span class="p">)</span><span class="w"> </span><span class="ow">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">RunDurationSec</span><span class="w"> </span><span class="nc">INT</span><span class="w"> </span><span class="ow">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">LogDateGMT</span><span class="w"> </span><span class="nc">DATETIME</span><span class="w"> </span><span class="ow">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">,</span><span class="w"> </span><span class="o">--</span><span class="w"> </span><span class="nc">Date</span><span class="w"> </span><span class="ow">and</span><span class="w"> </span><span class="nc">time</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">record</span><span class="w"> </span><span class="n">was</span><span class="w"> </span><span class="n">added</span><span class="w"></span>
<span class="w"> </span><span class="k">CONSTRAINT</span><span class="w"> </span><span class="n">PK_JobFailureArchive</span><span class="w"> </span><span class="k">PRIMARY</span><span class="w"> </span><span class="k">KEY</span><span class="w"> </span><span class="k">CLUSTERED</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">JobFailureArchiveID</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="k">FILLFACTOR</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="k">INDEX</span><span class="w"> </span><span class="n">IX_JobFailureArchive_RunDate</span><span class="w"> </span><span class="k">NONCLUSTERED</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">RunDate</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">JobName</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="k">FILLFACTOR</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="p">);</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
</code></pre></div>
<blockquote>
<p>Throughout the rest of this post I will be using the <code>DBA</code> database. This is a database I create on any instance I manage. I use it to store any objects I need to perform monitoring and troubleshooting tasks. I also tend to break out my objects by schema, in these examples I will be using the <code>Maintenance</code> schema.</p>
</blockquote>
<h2>The Procedure</h2>
<p>The procedure copies job failure data from the <code>dbo.sysjobhistory</code> table in <code>MSDB</code> to the <code>JobFailureArchive</code> table. Before it does this, the procedure scans the <code>JobFailureArchive</code> table to determine the date of the most recent job failure it had captured.</p>
<div class="codehilite"><pre><span></span><code><span class="n">USE</span><span class="w"> </span><span class="p">[</span><span class="n">DBA</span><span class="p">]</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="c1">----------------------------------------------------------------------------------</span>
<span class="c1">-- Procedure Name: Maintenance.ArchiveFailedJobs</span>
<span class="c1">--</span>
<span class="c1">-- Desc: Archives failed job details before purging job history.</span>
<span class="c1">--</span>
<span class="c1">-- Parameters:</span>
<span class="c1">--</span>
<span class="c1">-- Auth: Mark Wilkinson (@m82labs)</span>
<span class="c1">-- Date: 2017.08.20</span>
<span class="c1">----------------------------------------------------------------------------------</span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">OR</span><span class="w"> </span><span class="k">ALTER</span><span class="w"> </span><span class="k">PROCEDURE</span><span class="w"> </span><span class="p">[</span><span class="n">Maintenance</span><span class="p">].[</span><span class="n">ArchiveFailedJobs</span><span class="p">]</span><span class="w"></span>
<span class="k">AS</span><span class="w"></span>
<span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">lastFailure</span><span class="w"> </span><span class="n">DATETIME</span><span class="w"> </span><span class="c1">-- Last time a failure was captured</span>
<span class="c1">-- Get the most recent job failure recorded.</span>
<span class="k">SET</span><span class="w"> </span><span class="o">@</span><span class="n">lastFailure</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="k">SELECT</span><span class="w"></span>
<span class="w"> </span><span class="k">ISNULL</span><span class="p">(</span><span class="k">MAX</span><span class="p">(</span><span class="n">RunDate</span><span class="p">),</span><span class="s1">'19000101'</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">MaxRunDate</span><span class="w"></span>
<span class="w"> </span><span class="k">FROM</span><span class="w"></span>
<span class="w"> </span><span class="n">Maintenance</span><span class="p">.</span><span class="n">JobFailureArchive</span><span class="w"></span>
<span class="p">)</span><span class="w"></span>
<span class="c1">-- Insert new job failures</span>
<span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">Maintenance</span><span class="p">.</span><span class="n">JobFailureArchive</span><span class="w"></span>
<span class="p">(</span><span class="w"> </span><span class="n">JobID</span><span class="p">,</span><span class="w"> </span><span class="n">JobName</span><span class="p">,</span><span class="w"> </span><span class="n">RunDate</span><span class="p">,</span><span class="w"> </span><span class="n">StepID</span><span class="p">,</span><span class="w"> </span><span class="n">OutcomeText</span><span class="p">,</span><span class="w"> </span><span class="n">RunDuration</span><span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="k">SELECT</span><span class="w"></span>
<span class="w"> </span><span class="n">sj</span><span class="p">.</span><span class="n">job_id</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">sj</span><span class="p">.</span><span class="n">name</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">msdb</span><span class="p">.</span><span class="n">dbo</span><span class="p">.</span><span class="n">agent_datetime</span><span class="p">(</span><span class="n">jh</span><span class="p">.</span><span class="n">run_date</span><span class="p">,</span><span class="n">jh</span><span class="p">.</span><span class="n">run_time</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="n">jh</span><span class="p">.</span><span class="n">step_id</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">jh</span><span class="p">.</span><span class="n">message</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="n">jh</span><span class="p">.</span><span class="n">run_duration</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">10000</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">60</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">60</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="n">jh</span><span class="p">.</span><span class="n">run_duration</span><span class="w"> </span><span class="o">/</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="mi">60</span><span class="p">)</span><span class="w"> </span><span class="o">+</span><span class="w"></span>
<span class="w"> </span><span class="p">(</span><span class="n">jh</span><span class="p">.</span><span class="n">run_duration</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="mi">100</span><span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="k">As</span><span class="w"> </span><span class="n">RunDurationSec</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">GETUTCDATE</span><span class="p">()</span><span class="w"></span>
<span class="k">FROM</span><span class="w"></span>
<span class="w"> </span><span class="n">msdb</span><span class="p">.</span><span class="n">dbo</span><span class="p">.</span><span class="n">sysjobhistory</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">jh</span><span class="w"></span>
<span class="w"> </span><span class="k">INNER</span><span class="w"> </span><span class="k">JOIN</span><span class="w"> </span><span class="n">msdb</span><span class="p">.</span><span class="n">dbo</span><span class="p">.</span><span class="n">sysjobs</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">sj</span><span class="w"></span>
<span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="n">jh</span><span class="p">.</span><span class="n">job_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">sj</span><span class="p">.</span><span class="n">job_id</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"></span>
<span class="w"> </span><span class="n">jh</span><span class="p">.</span><span class="n">run_status</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="c1">-- Just get failures</span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">jh</span><span class="p">.</span><span class="n">step_id</span><span class="w"> </span><span class="o"><></span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="c1">-- Skip the '0' step that gives us summary info</span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">msdb</span><span class="p">.</span><span class="n">dbo</span><span class="p">.</span><span class="n">agent_datetime</span><span class="p">(</span><span class="n">jh</span><span class="p">.</span><span class="n">run_date</span><span class="p">,</span><span class="n">jh</span><span class="p">.</span><span class="n">run_time</span><span class="p">)</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="o">@</span><span class="n">lastFailure</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<h2>Alerting</h2>
<p>How you approach alerting is up to you, but you have a lot of different options. In my environment I query the <code>JobFailureArchive</code> table via a Nagios service check. You could do the same thing with any monitoring solution that allows you to add custom queries. No matter which direction you take, writing a query to get the job failure details is simple:</p>
<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span><span class="w"> </span><span class="n">TOP</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="n">TIES</span><span class="w"></span>
<span class="w"> </span><span class="n">JobName</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">Rundate</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">StepID</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">OutcomeText</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">RunDurationSec</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">DBA</span><span class="p">.</span><span class="n">Maintenance</span><span class="p">.</span><span class="n">JobFailureArchive</span><span class="w"></span>
<span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">LogDateGMT</span><span class="w"></span>
</code></pre></div>
<p>Since we are storing the date the records are added to the table, this query will always return the latest set of failures. This is a simple example, but the possibilities are endless:
* Send the results of this query via database mail
* Join with <code>dbo.sysjobs</code> and <code>dbo.syscategories</code>, alerting on different thresholds per job category
* Extend the <code>TOP (1)</code> to include multiple capture periods and alert on average failures per capture </p>
<h2>Configuring the SQL Agent Job</h2>
<p>A SQL Agent job is the last component that ties everything together. At it's simplest the job executes the procedure in the example above. This will get your job failures into the archive table, but there is more this job can do.</p>
<p>Since we are archiving job failures, we can clean up job history in MSDB more aggressively. To do this, add another step to the agent job that executes the following (which limits job history to a rolling 24 hours):</p>
<div class="codehilite"><pre><span></span><code><span class="k">DECLARE</span><span class="w"> </span><span class="o">@</span><span class="n">OldestJob</span><span class="w"> </span><span class="n">DATETIME</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">DATEADD</span><span class="p">(</span><span class="k">day</span><span class="p">,</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="n">GETDATE</span><span class="p">())</span><span class="w"></span>
<span class="k">EXEC</span><span class="w"> </span><span class="n">msdb</span><span class="p">.</span><span class="n">dbo</span><span class="p">.</span><span class="n">sp_purge_jobhistory</span><span class="w"> </span><span class="o">@</span><span class="n">oldest_date</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">@</span><span class="n">OldestJob</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>Reducing the size of the <code>sysjobhistory</code> table can have a lot of benefits including: quicker restore times on MSDB, more responsive SSMS when viewing job history, and a reduction in contention when writing to a busy <code>sysjobhistory</code> table.</p>
<p>You could also add a step to send an email alert when new job failures are inserted. Database mail is a rabbit hole we aren't going to explore in this post. If you do plan on using this job to alert, start by reading through the resources at the bottom of this page. Specifically the one by Jes Borland on sending query results using database mail.</p>
<p>The article on using tokens in agent jobs is also a good read. Using tokens you can find job failures inserted since the job started executing:</p>
<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span><span class="w"> </span><span class="n">JobName</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">Rundate</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">StepID</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">OutcomeText</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">RunDurationSec</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">DBA</span><span class="p">.</span><span class="n">Maintenance</span><span class="p">.</span><span class="n">JobFailureArchive</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"> </span><span class="n">LogDateGMT</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="n">msdb</span><span class="p">.</span><span class="n">dbo</span><span class="p">.</span><span class="n">agent_datetime</span><span class="p">(</span><span class="err">$</span><span class="p">(</span><span class="n">ESCAPE_SQUOTE</span><span class="p">(</span><span class="n">STRTDT</span><span class="p">)),</span><span class="w"> </span><span class="err">$</span><span class="p">(</span><span class="n">ESCAPE_SQUOTE</span><span class="p">(</span><span class="n">STRTTM</span><span class="p">)))</span><span class="w"></span>
<span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">LogDateGMT</span><span class="w"></span>
</code></pre></div>
<p>The system function <code>msdb.dbo.agent_datetime</code> converts the date and time format used by SQL Agent into a standard datetime. The tokens <code>$(ESCAPE_SQUOTE(STRTDT))</code> and <code>$(ESCAPE_SQUOTE(STRTTM))</code>, will return the date and time the job started executing. Combined with Jes's post this could be all you need for job failure alerts.</p>
<blockquote>
<p>If this job fails for any reason you will stop receiving notifications for <strong>any</strong> job failures. Make sure to configure this job to alert via email if it fails! </p>
</blockquote>
<h2>PowerShell (Bonus Feature!)</h2>
<p>One nice thing about having job failure data in a dedicated table is that it is easy to query. You can also give normal users access to the data without having to put them in special agent roles or granting them access to system databases. Below is a PowerShell function you, or your users, could use to retrieve job failures based on several criteria:</p>
<div class="codehilite"><pre><span></span><code><span class="k">function</span> <span class="nb">Get-SqlJobFailures</span>
<span class="p">{</span>
<span class="cm"><#</span>
<span class="cm"> </span><span class="sd">.SYNOPSIS</span><span class="cm"></span>
<span class="cm"> Retrieves job failure details from the given instance</span>
<span class="cm"> </span><span class="sd">.PARAMETER</span><span class="cm"> SqlInstance (Accepts values from the pipeline)</span>
<span class="cm"> Instance we are getting job failures from</span>
<span class="cm"> </span><span class="sd">.PARAMETER</span><span class="cm"> JobName</span>
<span class="cm"> All or part of the job name you are interested in</span>
<span class="cm"> </span><span class="sd">.PARAMETER</span><span class="cm"> Newest</span>
<span class="cm"> The number of results to return based on date</span>
<span class="cm"> </span><span class="sd">.PARAMETER</span><span class="cm"> Since</span>
<span class="cm"> The oldest date to show in the results</span>
<span class="cm"> </span><span class="sd">.PARAMETER</span><span class="cm"> SearchString</span>
<span class="cm"> This parameter can be used to search the error output</span>
<span class="cm"> #></span>
<span class="p">[</span><span class="k">CmdletBinding</span><span class="p">()]</span>
<span class="k">param</span><span class="p">(</span>
<span class="p">[</span><span class="k">Parameter</span><span class="p">(</span><span class="k">Mandatory</span><span class="p">=</span><span class="nv">$true</span><span class="p">,</span><span class="k">ValueFromPipeline</span><span class="p">=</span><span class="nv">$true</span><span class="p">)]</span>
<span class="no">[string]</span><span class="nv">$SqlInstance</span><span class="p">,</span>
<span class="no">[string]</span><span class="nv">$JobName</span> <span class="p">=</span> <span class="s1">''</span><span class="p">,</span>
<span class="no">[int]</span><span class="nv">$Newest</span> <span class="p">=</span> <span class="n">10000</span><span class="p">,</span>
<span class="no">[datetime]</span><span class="nv">$Since</span> <span class="p">=</span> <span class="s1">'2017-01-01'</span><span class="p">,</span>
<span class="no">[string]</span><span class="nv">$SearchString</span>
<span class="p">)</span>
<span class="k">BEGIN</span>
<span class="p">{</span>
<span class="no">[string]</span><span class="nv">$GetFailures_Query</span> <span class="p">=</span> <span class="sh">@"</span>
<span class="sh"> SELECT TOP(</span><span class="p">$(</span><span class="nv">$Newest</span><span class="p">)</span><span class="sh">)</span>
<span class="sh"> @@SERVERNAME AS ServerName,</span>
<span class="sh"> JobName,</span>
<span class="sh"> Rundate,</span>
<span class="sh"> StepID,</span>
<span class="sh"> OutcomeText,</span>
<span class="sh"> RunDurationSec</span>
<span class="sh"> FROM DBA.Maintenance.JobFailureArchive</span>
<span class="sh"> WHERE rundate >= '</span><span class="p">$(</span><span class="nv">$Since</span><span class="p">.</span><span class="n">ToString</span><span class="p">(</span><span class="s1">'yyyMMdd HH:mm'</span><span class="p">))</span><span class="sh">'</span>
<span class="sh"> AND JobName LIKE '%</span><span class="p">$(</span><span class="nv">$JobName</span><span class="p">)</span><span class="sh">%'</span>
<span class="sh"> </span><span class="p">$(</span> <span class="k">if</span><span class="p">(</span><span class="nv">$SearchString</span><span class="p">){</span> <span class="s2">"AND OutcomeText LIKE '%</span><span class="p">$(</span><span class="nv">$SearchString</span><span class="p">)</span><span class="s2">%'"</span> <span class="p">})</span><span class="sh"></span>
<span class="sh"> ORDER BY RunDate DESC;</span>
<span class="sh">"@</span>
<span class="nv">$data</span> <span class="p">=</span> <span class="p">@()</span>
<span class="p">}</span>
<span class="k">PROCESS</span>
<span class="p">{</span>
<span class="nb">Write-Host</span> <span class="s2">"Testing connection to </span><span class="p">$(</span><span class="nv">$SqlInstance</span><span class="p">)</span><span class="s2">: "</span> <span class="n">-NoNewline</span>
<span class="k">try</span>
<span class="p">{</span>
<span class="nb">Test-Connection</span> <span class="n">-ComputerName</span> <span class="nv">$SqlInstance</span> <span class="n">-Count</span> <span class="n">1</span> <span class="n">-Quiet</span> <span class="n">-ErrorAction</span> <span class="n">Stop</span> <span class="p">|</span> <span class="nb">Out-Null</span>
<span class="nb">Write-Host</span> <span class="s2">"success"</span> <span class="n">-ForegroundColor</span> <span class="n">Green</span>
<span class="p">}</span>
<span class="k">catch</span>
<span class="p">{</span>
<span class="nb">Write-Host</span> <span class="s2">"error - </span><span class="p">$(</span><span class="nv">$_</span><span class="p">.</span><span class="n">Exception</span><span class="p">.</span><span class="n">Message</span><span class="p">)</span><span class="s2">"</span> <span class="n">-ForegroundColor</span> <span class="n">Red</span>
<span class="k">throw</span>
<span class="p">}</span>
<span class="nb">Write-Host</span> <span class="s2">"Getting job failure data: "</span> <span class="n">-NoNewline</span>
<span class="k">try</span>
<span class="p">{</span>
<span class="nb">Invoke-Sqlcmd</span> <span class="n">-Query</span> <span class="nv">$GetFailures_Query</span> <span class="n">-ServerInstance</span> <span class="nv">$SqlInstance</span> <span class="n">-Database</span> <span class="n">master</span> <span class="p">|</span> <span class="p">%</span> <span class="p">{</span>
<span class="nv">$data</span> <span class="p">+=</span> <span class="nv">$_</span>
<span class="p">}</span>
<span class="nb">Write-Host</span> <span class="s2">"done"</span> <span class="n">-ForegroundColor</span> <span class="n">Green</span>
<span class="p">}</span>
<span class="k">catch</span>
<span class="p">{</span>
<span class="nb">Write-Host</span> <span class="s2">"error - </span><span class="p">$(</span><span class="nv">$_</span><span class="p">.</span><span class="n">Exception</span><span class="p">.</span><span class="n">Message</span><span class="p">)</span><span class="s2">"</span> <span class="n">-ForegroundColor</span> <span class="n">Red</span>
<span class="k">throw</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">END</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$data</span> <span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="p">$(</span><span class="nv">$data</span> <span class="p">|</span> <span class="nb">Sort-Object</span> <span class="n">-Property</span> <span class="n">ServerName</span><span class="p">,</span> <span class="n">RunDate</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>If you are not familiar with PowerShell I recommend giving it a try. It has become an indispensable tool for my day to day tasks. The power of this function comes when you run it across multiple instances:</p>
<div class="codehilite"><pre><span></span><code><span class="n">PS</span><span class="p">></span> <span class="p">@(</span><span class="s1">'sql-instance-01'</span><span class="p">,</span> <span class="p">`</span>
<span class="s1">'sql-instance-02'</span><span class="p">,</span> <span class="p">`</span>
<span class="s1">'sql-instance-03'</span> <span class="p">)</span> <span class="p">|</span> <span class="nb">Get-SqlJobFailures</span> <span class="n">-Newest</span> <span class="n">15</span> <span class="p">|</span> <span class="nb">Out-Gridview</span>
</code></pre></div>
<p>This will get the 15 most recent job failures for the three instances listed, and display them in a graphical grid view.</p>
<h1>Final Thoughts</h1>
<p>Writing job failures to a user table seems like a simple idea, but you'll be surprised how much you use it. We only scratched the surface in this post, I would love to hear what other uses you find for this. </p>
<h1>Resources</h1>
<ul>
<li><a href="https://docs.microsoft.com/en-us/sql/ssms/agent/use-tokens-in-job-steps">Use Tokens in Job Steps</a> - Microsoft Docs</li>
<li><a href="https://www.brentozar.com/archive/2014/10/send-query-results-sql-server-agent-job/">Email Query Results Using a SQL Server Agent Job</a> - Jes Borland via BrentOzar.com</li>
<li><a href="https://www.nagios.com/">Nagios</a> - Nagios Home Page</li>
</ul>automation basics with powershell and cms2017-09-12cms-poshRetrieving servers from a Central Management Server using PowerShell.<p>As a DBA I use PowerShell every day. If it can be automated, I will automate it. In my environment I have 300+ instances to keep track of and maintain. When you manage that many instances, it's important to have a "source of truth" for a full list of your instances. This is where Central Management Server (CMS) comes in. The combination of PowerShell and CMS is a powerful one. In this post we'll explore a PowerShell function I wrote to retrieve servers from my CMS server.</p>
<p>I want to thank Rob Sewell (<a href="https://sqldbawithabeard.com/">WWW</a>/<a href="https://twitter.com/sqldbawithbeard">Twitter</a>) for hosting this months T-SQL Tuesday! If you are interested in learning more about T-SQL Tuesday, take a look at Rob's <a href="https://sqldbawithabeard.com/2017/09/05/tsql2sday-94-lets-get-all-posh/">post</a>, and check out the <a href="https://twitter.com/hashtag/tsql2sday?f=realtime&src=hash">#tsql2sday</a> hashtag on Twitter.</p>
<blockquote>
<p>Central Management Server is an old feature, and easy to configure in Management Studio. CMS allows you to create a list of your instances on a central server. You can group your instances and, in Management Studio, run queries across multiple instances at a time. To learn more about CMS, head over to: <a href="https://docs.microsoft.com/en-us/sql/ssms/register-servers/create-a-central-management-server-and-server-group">Microsoft Docs</a>.</p>
</blockquote>
<h1>Get-CmsHosts</h1>
<p><code>Get-CmsHosts</code> is a function I wrote as part of a custom PowerShell module we maintain internally at my employer. It is simple to use, but is the base of most automation projects I work on. </p>
<h1>Simple Example</h1>
<div class="codehilite"><pre><span></span><code><span class="n">PS</span><span class="p">></span> <span class="nb">Get-CmsHosts</span> <span class="n">-SqlInstance</span> <span class="s1">'srv-'</span> <span class="n">-CmsInstance</span> <span class="n">srv-mycms</span><span class="p">-</span><span class="n">01</span>
</code></pre></div>
<p>This example will connect to <code>srv-mycms-01</code> and return a distinct list of instance host names registered with that CMS server that start with the string <code>srv-</code>. This output can then be piped to other commands:</p>
<div class="codehilite"><pre><span></span><code><span class="n">PS</span><span class="p">></span> <span class="nb">Get-CmsHosts</span> <span class="n">-SqlInstance</span> <span class="s1">'srv-'</span> <span class="n">-CmsInstance</span> <span class="n">srv-mycms</span><span class="p">-</span><span class="n">01</span> <span class="p">|</span> <span class="p">%</span> <span class="p">{</span>
<span class="nb">Invoke-SqlCmd</span> <span class="n">-Query</span> <span class="s1">'exec dbo.MyProc'</span> <span class="n">-ServerInstance</span> <span class="nv">$_</span>
<span class="p">}</span>
</code></pre></div>
<p>In this example <code>%</code> is short-hand for <code>ForEach</code>. This will iterate through the list of instances and store the name of the current instance in a temporary variable <code>$_</code>. We then use that variable to run <code>Invoke-SqlCmd</code>. You can already see how useful this can be.</p>
<h1>Specific Versions</h1>
<p>It is common to use this function when we are remotely upgrading instances. In this specific use case it is very important that you only run the upgrade or patch on SQL Server running a specific version. </p>
<div class="codehilite"><pre><span></span><code><span class="n">PS</span><span class="p">></span> <span class="nb">Get-CmsHosts</span> <span class="n">-Version</span> <span class="s1">'13.0.1605.1'</span> <span class="n">-CmsInstance</span> <span class="n">srv-mycms</span><span class="p">-</span><span class="n">01</span>
</code></pre></div>
<p>This command will only return instances running version 13.0.1605.1 (2016 RTM) of SQL Server.</p>
<blockquote>
<p>When using <code>-Version</code>, try to filter using the name as well. In order to get the version of the instance the function will connect to each one and run a query. If you have a lot of instances, this will take a lot of time.</p>
</blockquote>
<h1>List of Databases</h1>
<p>Sometimes you might need to run a query of PowerShell cmdlet against all of the databases in on a given server, or group of servers. Using the <code>-EnumDatabases</code> you can do this:</p>
<div class="codehilite"><pre><span></span><code><span class="n">PS</span><span class="p">></span> <span class="nb">Get-CmsHosts</span> <span class="n">-SqlInstance</span> <span class="n">srv-server</span><span class="p">-</span><span class="n">01</span> <span class="n">-CmsInstance</span> <span class="n">srv-mycms</span><span class="p">-</span><span class="n">01</span> <span class="n">-EnumDatabases</span> <span class="p">|</span> <span class="p">%</span> <span class="p">{</span>
<span class="nv">$CurrServer</span> <span class="p">=</span> <span class="nv">$_</span>
<span class="nv">$CurrDatabases</span> <span class="p">=</span> <span class="nv">$_</span><span class="p">.</span><span class="n">Databases</span>
<span class="nv">$CurrDatabases</span> <span class="p">|</span> <span class="p">%</span> <span class="p">{</span>
<span class="nb">Invoke-SqlCmd</span> <span class="n">-Query</span> <span class="s1">'exec dbo.MyProc'</span> <span class="n">-Database</span> <span class="nv">$_</span> <span class="n">-ServerInstance</span> <span class="nv">$CurrServer</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p><code>EnumDatabases</code> will enumerate the databases on any instances that are returned. In the above example we are running <code>dbo.MyProc</code> against each instance, and each database on that instance, returned by the call to Cms-Hosts. When using <code>EnumDatabases</code> the list of databases on the instance is queried and returned with the instance. This means each instance returned by the function will have two members. One named <code>Name</code> that holds the name of the instance. And another named <code>Databases</code> that holds the list of databases. Besides upgrades, this can also be useful for tasks like setting database owners, or checking/applying a security configuration.</p>
<h1>Configuration and Other Options</h1>
<p>You can avoid having to pass in the <code>CmsInstance</code> parameter if you simply set an OS level environment variable named <code>SQL_CMS</code> that holds the name of your CMS instance. If you want to test this, the first example could be re-written to shows it's use of the environment variable:</p>
<div class="codehilite"><pre><span></span><code><span class="n">PS</span><span class="p">></span> <span class="nv">$env:SQL_CMS</span> <span class="p">=</span> <span class="s1">'srv-mycms-01'</span>
<span class="n">PS</span><span class="p">></span> <span class="nb">Get-CmsHosts</span> <span class="n">-SqlInstance</span> <span class="s1">'srv-'</span>
</code></pre></div>
<p>While CMS can be very useful, it isn't always available, especially if you are testing your scripts on your local computer while disconnected from the network. For cases like this, and to keep your scripts consistent, you can pass a file path to the <code>SqlInstance</code> parameter. the file should contain a single instance name per line. If the function sees that a file was passed in, it will use the file contents instead of connecting to the CMS. All the other parameters still work as expected.</p>
<h1>Get the Script!</h1>
<p>PowerShell and CMS are a combination I use for almost all of my automation tasks. Grab a copy of the function and see what you can use it for. If you have any suggestions, or want to submit updates, please leave comments here or issue a pull request on GitHub.</p>
<p>You can get your copy of Get-CmsHosts at my Github repo: <a href="https://raw.githubusercontent.com/m82labs/post_scripts/master/PowerShell/Get-CMSHosts.ps1">Post Scripts</a> </p>a closer look at output2018-01-15closer-look-outputThe many uses of the TSQL OUTPUT clause.<p>Lots of people have used the <code>OUTPUT</code> clause in an <code>INSERT</code>/<code>UPDATE</code>/<code>DELETE</code> statement, but many may not know just how flexible it is, and what hidden gems there are in the documentation. In this post we are going to look at two interesting and surprising use cases for the <code>OUTPUT</code> clause.</p>
<h2>The Basics</h2>
<p>For those of you that might not know how <code>OUTPUT</code> works, let's walk through a quick demo:</p>
<div class="codehilite"><pre><span></span><code><span class="k">CREATE</span><span class="w"> </span><span class="k">DATABASE</span><span class="w"> </span><span class="n">OutputDemo</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="n">USE</span><span class="w"> </span><span class="n">OutputDemo</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="c1">-- Big table of GUIDs</span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">MyGuid</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="n">MyGuidID</span><span class="w"> </span><span class="nb">INT</span><span class="w"> </span><span class="k">IDENTITY</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">),</span><span class="w"></span>
<span class="n">GUID</span><span class="w"> </span><span class="n">UNIQUEIDENTIFIER</span><span class="w"></span>
<span class="p">);</span><span class="w"></span>
<span class="c1">-- GUID change log</span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">MyGuid_Log</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="n">MyGuid_LogID</span><span class="w"> </span><span class="nb">INT</span><span class="w"> </span><span class="k">IDENTITY</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">),</span><span class="w"></span>
<span class="n">MyGuidID</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span><span class="w"></span>
<span class="n">GUID</span><span class="w"> </span><span class="n">UNIQUEIDENTIFIER</span><span class="p">,</span><span class="w"></span>
<span class="n">ChangeDate</span><span class="w"> </span><span class="n">DATETIME</span><span class="w"> </span><span class="k">DEFAULT</span><span class="w"> </span><span class="n">GETUTCDATE</span><span class="p">()</span><span class="w"></span>
<span class="p">);</span><span class="w"></span>
</code></pre></div>
<p>In the script we create a test database, create a table <code>MyGuid</code> to store some data, and also a log table <code>MyGuid_Log</code> to store historical records on what was changed in our table. At this point you have a few options to get the log table populated. You could implement a trigger on <code>MyGuid</code> that inserts records into the log, but you could also use the <code>OUTPUT</code> clause:</p>
<div class="codehilite"><pre><span></span><code><span class="c1">-- Insert 10000 records, log the records in the log table</span>
<span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">MyGuid</span><span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="n">GUID</span><span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="c1">--== Insert the records into the log table as well</span>
<span class="k">OUTPUT</span><span class="w"> </span><span class="n">Inserted</span><span class="p">.</span><span class="o">*</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">MyGuid_Log</span><span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="n">MyGuidID</span><span class="p">,</span><span class="n">GUID</span><span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="k">SELECT</span><span class="w"> </span><span class="n">TOP</span><span class="p">(</span><span class="mi">10000</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">NEWID</span><span class="p">()</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">sys</span><span class="p">.</span><span class="n">all_columns</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">c1</span><span class="w"></span>
<span class="w"> </span><span class="k">CROSS</span><span class="w"> </span><span class="k">JOIN</span><span class="w"> </span><span class="n">sys</span><span class="p">.</span><span class="n">all_columns</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">c2</span><span class="p">;</span><span class="w"></span>
<span class="c1">-- Update records 100-1000</span>
<span class="k">UPDATE</span><span class="w"> </span><span class="n">MyGuid</span><span class="w"></span>
<span class="w"> </span><span class="k">SET</span><span class="w"> </span><span class="n">GUID</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">NEWID</span><span class="p">()</span><span class="w"></span>
<span class="c1">--== Insert the newly updated records into the log table</span>
<span class="k">OUTPUT</span><span class="w"> </span><span class="n">Inserted</span><span class="p">.</span><span class="o">*</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">MyGuid_Log</span><span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="n">MyGuidID</span><span class="p">,</span><span class="n">GUID</span><span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"> </span><span class="n">MyGuidID</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="mi">100</span><span class="w"></span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">MyGuidID</span><span class="w"> </span><span class="o"><=</span><span class="w"> </span><span class="mi">1000</span><span class="p">;</span><span class="w"></span>
<span class="c1">-- Delete Records 900-1000</span>
<span class="k">DELETE</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">MyGuid</span><span class="w"></span>
<span class="c1">--== Insert the newly updated records into the log table</span>
<span class="k">OUTPUT</span><span class="w"> </span><span class="n">Deleted</span><span class="p">.</span><span class="o">*</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">MyGuid_Log</span><span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="n">MyGuidID</span><span class="p">,</span><span class="n">GUID</span><span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"> </span><span class="n">MyGuid</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="mi">900</span><span class="w"></span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">MyGuid</span><span class="w"> </span><span class="o"><=</span><span class="w"> </span><span class="mi">1000</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>Based on this code, all 10,000 inserts will also be inserted into the <code>MyGuid_Log</code> table as well as the newly updated records. We are accessing the new/updated records using the <code>Inserted</code> psuedo-table provided by the <code>OUTPUT</code> clause. This is similar to how you would interact with records in a trigger. When updating and deleting records you also have access to the <code>Deleted</code> psuedo-table, allowing you to log both the current and previous state of a record if desired. It's also important to note that you can insert into a table variable, or remove the <code>INTO</code> portion of the statement completely and just return the output as a result set to be consumed.</p>
<h1>"Tee" Time</h1>
<p>A common command in the Linux world is the <code>tee</code> command. What <code>tee</code> allows you to do is pipe the output of a command to a file as well as the console. This same functionality can be implemented using multiple <code>OUTPUT</code> clauses in a T-SQL statement. In this example we are going to update a few hundred records. When the update statement is run, not only will it update the MyGuid table but it will update a log table and also return the result of the update. This is accomplished by using two <code>OUTPUT</code> clauses.</p>
<div class="codehilite"><pre><span></span><code><span class="n">USE</span><span class="w"> </span><span class="n">OutputDemo</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="c1">-- Update records 100-1000</span>
<span class="k">UPDATE</span><span class="w"> </span><span class="n">MyGuid</span><span class="w"></span>
<span class="w"> </span><span class="k">SET</span><span class="w"> </span><span class="n">GUID</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">NEWID</span><span class="p">()</span><span class="w"></span>
<span class="c1">--== Insert records into a log table</span>
<span class="k">OUTPUT</span><span class="w"> </span><span class="n">Inserted</span><span class="p">.</span><span class="o">*</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">MyGuid_Log</span><span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="n">MyGuidID</span><span class="p">,</span><span class="w"> </span><span class="n">GUID</span><span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="c1">--== Also return the updated records as a result set</span>
<span class="k">OUTPUT</span><span class="w"> </span><span class="n">Inserted</span><span class="p">.</span><span class="o">*</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"> </span><span class="n">MyGuidID</span><span class="w"> </span><span class="o">>=</span><span class="w"> </span><span class="mi">100</span><span class="w"></span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">MyGuidID</span><span class="w"> </span><span class="o"><=</span><span class="w"> </span><span class="mi">1000</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>This can be useful in cases where you need to do some logging but also need to know what records were just altered by a statement. In my example I am selecting <code>*</code> from <code>Inserted</code>, but you could just as easily select the ID column only (or any column you need), and return that to the app making the call. </p>
<h1>SELECT FROM ... UPDATE?</h1>
<p>While the "tee" functionality is useful, this next behavior took me by surprise. In this example we are going to illustrate how to nest an <code>UPDATE</code> statement <em>within</em> an <code>INSERT</code> statement as a sub-query. This feature of the <code>OUTPUT</code> clause is discussed in Microsoft Docs, but because they chose to illustrate the concept using a <code>MERGE</code> statement, it isn't immediately obvious what it's doing. In this script we are going to be working with some product inventory information. We will create an <code>ItemQty</code> table to store item quantity information, an <code>Item</code> table that store item details (just a name in this case), and an <code>Item_History</code> table to store changes to an item.</p>
<div class="codehilite"><pre><span></span><code><span class="n">USE</span><span class="w"> </span><span class="n">OutputDemo</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">Item</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">ItemID</span><span class="w"> </span><span class="nb">INT</span><span class="w"> </span><span class="k">IDENTITY</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="n">ItemName</span><span class="w"> </span><span class="n">NVARCHAR</span><span class="p">(</span><span class="mi">128</span><span class="p">)</span><span class="w"></span>
<span class="p">)</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">ItemQty</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">ItemID</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">Qty</span><span class="w"> </span><span class="nb">INT</span><span class="w"></span>
<span class="p">);</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">ItemQty_History</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">ItemId</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">ItemName</span><span class="w"> </span><span class="n">NVARCHAR</span><span class="p">(</span><span class="mi">128</span><span class="p">),</span><span class="w"></span>
<span class="w"> </span><span class="n">OldQty</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">NewQty</span><span class="w"> </span><span class="nb">INT</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">ChangeDate</span><span class="w"> </span><span class="n">DATETIME</span><span class="w"> </span><span class="k">DEFAULT</span><span class="p">(</span><span class="n">GETUTCDATE</span><span class="p">())</span><span class="w"></span>
<span class="p">);</span><span class="w"></span>
</code></pre></div>
<p>Now we are going to populate our Item table with a few random string values (GUIDs), and then we will insert some item quantity data based on these items. We are going to use a "tally table" in this example just to generate some rows.</p>
<div class="codehilite"><pre><span></span><code><span class="k">WITH</span><span class="w"> </span><span class="n">lv0</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="k">g</span><span class="w"> </span><span class="k">UNION</span><span class="w"> </span><span class="k">ALL</span><span class="w"> </span><span class="k">SELECT</span><span class="w"> </span><span class="mi">0</span><span class="p">)</span><span class="w"></span>
<span class="p">,</span><span class="n">lv1</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="k">g</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">lv0</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="k">CROSS</span><span class="w"> </span><span class="k">JOIN</span><span class="w"> </span><span class="n">lv0</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="c1">-- 4</span>
<span class="p">,</span><span class="n">lv2</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="k">g</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">lv1</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="k">CROSS</span><span class="w"> </span><span class="k">JOIN</span><span class="w"> </span><span class="n">lv1</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="c1">-- 16</span>
<span class="p">,</span><span class="n">lv3</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="k">g</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">lv2</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="k">CROSS</span><span class="w"> </span><span class="k">JOIN</span><span class="w"> </span><span class="n">lv2</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="c1">-- 256</span>
<span class="p">,</span><span class="n">lv4</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="k">g</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">lv3</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="k">CROSS</span><span class="w"> </span><span class="k">JOIN</span><span class="w"> </span><span class="n">lv3</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="c1">-- 65,536</span>
<span class="p">,</span><span class="n">lv5</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="mi">0</span><span class="w"> </span><span class="k">g</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">lv4</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="k">CROSS</span><span class="w"> </span><span class="k">JOIN</span><span class="w"> </span><span class="n">lv4</span><span class="w"> </span><span class="n">b</span><span class="p">)</span><span class="w"> </span><span class="c1">-- 4,294,967,296</span>
<span class="p">,</span><span class="n">Tally</span><span class="w"> </span><span class="p">(</span><span class="n">n</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="n">ROW_NUMBER</span><span class="p">()</span><span class="w"> </span><span class="n">OVER</span><span class="w"> </span><span class="p">(</span><span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="p">(</span><span class="k">SELECT</span><span class="w"> </span><span class="k">NULL</span><span class="p">))</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">lv5</span><span class="p">)</span><span class="w"></span>
<span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">Item</span><span class="w"> </span><span class="p">(</span><span class="n">ItemName</span><span class="p">)</span><span class="w"></span>
<span class="k">SELECT</span><span class="w"> </span><span class="n">TOP</span><span class="p">(</span><span class="mi">1000</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="n">NEWID</span><span class="p">()</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">Tally</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">ItemQty</span><span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="n">ItemID</span><span class="p">,</span><span class="w"> </span><span class="n">Qty</span><span class="p">)</span><span class="w"> </span>
<span class="k">SELECT</span><span class="w"> </span><span class="n">ItemID</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="k">ABS</span><span class="p">(</span><span class="n">BINARY_CHECKSUM</span><span class="p">(</span><span class="n">NEWID</span><span class="p">()))</span><span class="w"> </span><span class="o">%</span><span class="w"> </span><span class="mi">1000</span><span class="w"> </span><span class="c1">--= Generate a random number between 0-999</span>
<span class="k">FROM</span><span class="w"> </span><span class="n">Item</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
</code></pre></div>
<p>So far we have an <code>Item</code> table full of some fake items, and an <code>ItemQty</code> table full of item quantity values. Now we want to run a statement to update quantity, but we need to make sure that change gets logged in our history table. When we created the history table we included the <code>Item.ItemName</code> column, as well as the old and new values for <code>ItemQty.Qty</code>. Using a single T-SQL statement, and no triggers, we can update quantity while also inserting data from two different tables into a history table. Not only are we going to combine data from two tables, but we'll also be making sure we are only inserting data when the quantity was actually changed. If the old and new quantity values are the same, nothing will be inserted.</p>
<div class="codehilite"><pre><span></span><code><span class="k">INSERT</span><span class="w"> </span><span class="k">INTO</span><span class="w"> </span><span class="n">dbo</span><span class="p">.</span><span class="n">ItemQty_History</span><span class="w"></span>
<span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">ItemId</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">ItemName</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">OldQty</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">NewQty</span><span class="w"></span>
<span class="p">)</span><span class="w"></span>
<span class="k">SELECT</span><span class="w"> </span><span class="n">NewData</span><span class="p">.</span><span class="n">ItemId</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">NewData</span><span class="p">.</span><span class="n">ItemName</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">NewData</span><span class="p">.</span><span class="n">OldQty</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">NewData</span><span class="p">.</span><span class="n">NewQty</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="c1">--== Treat an UPDATE as a sub-query, using OUTPUT</span>
<span class="w"> </span><span class="k">UPDATE</span><span class="w"> </span><span class="n">iq</span><span class="w"></span>
<span class="w"> </span><span class="k">SET</span><span class="w"> </span><span class="n">Qty</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="w"> </span><span class="k">OUTPUT</span><span class="w"> </span><span class="n">Inserted</span><span class="p">.</span><span class="n">ItemID</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">Item</span><span class="p">.</span><span class="n">ItemName</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">Deleted</span><span class="p">.</span><span class="n">Qty</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">OldQty</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">Inserted</span><span class="p">.</span><span class="n">Qty</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">NewQty</span><span class="w"></span>
<span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">dbo</span><span class="p">.</span><span class="n">ItemQty</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">iq</span><span class="w"></span>
<span class="w"> </span><span class="k">INNER</span><span class="w"> </span><span class="k">JOIN</span><span class="w"> </span><span class="n">dbo</span><span class="p">.</span><span class="n">Item</span><span class="w"></span>
<span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="n">iq</span><span class="p">.</span><span class="n">ItemID</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">Item</span><span class="p">.</span><span class="n">ItemID</span><span class="w"></span>
<span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">Qty</span><span class="w"> </span><span class="o">></span><span class="w"> </span><span class="mi">100</span><span class="w"></span>
<span class="w"> </span><span class="c1">--== Now we alias the OUTPUT as 'NewData'</span>
<span class="w"> </span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">NewData</span><span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="n">ItemId</span><span class="p">,</span><span class="w"> </span><span class="n">ItemName</span><span class="p">,</span><span class="w"> </span><span class="n">OldQty</span><span class="p">,</span><span class="w"> </span><span class="n">NewQty</span><span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="k">WHERE</span><span class="w"> </span><span class="n">NewData</span><span class="p">.</span><span class="n">OldQty</span><span class="w"> </span><span class="o"><></span><span class="w"> </span><span class="n">NewData</span><span class="p">.</span><span class="n">NewQty</span><span class="p">;</span><span class="w"> </span><span class="c1">--== A where clause applied to the output of the UPDATE</span>
</code></pre></div>
<p>Since we are treating the output of the <code>UPDATE</code> statement as a table, we can filter that output, making sure we only get records that have changed. If we look at our history table, we should now see records where the quantity was changed:</p>
<div class="codehilite"><pre><span></span><code><span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">dbo</span><span class="p">.</span><span class="n">ItemQty_History</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>One of the more interesting things about this method is the query plan. Based on some analysis using live query stats, the plan seems to stream the records into the history table as the updates occur. There are no intermediate objects created here, so unless your use case forces a merge join or another operation requiring sorting, your tempdb usage should be minimal. Here is the plan on my system (using SQL Operations Studio running against a SQL on Linux container):</p>
<p><img alt="OUTPUT Query Plan" src="/img/OUTPUT-QueryPlan.png" /></p>
<h1>Final Thoughts</h1>
<p>Like anything else in SQL Server, you need to weigh the benefits of using any of the code above with how unfamiliar it might be to people that have to maintain it. With proper commenting you should be able to make it clear what you are trying to accomplish. There are a few restrictions when using <code>OUTPUT</code>, so I would suggest reading through the <a href="https://docs.microsoft.com/en-us/sql/t-sql/queries/output-clause-transact-sql">full documentation</a> to make sure you fully understand what you are getting into. Along with a full explanation of the restrictions, there are also a few other neat use cases proposed, like a queue implemented using <code>DELETE TOP(1)</code> and <code>OUTPUT</code>. </p>bash, an introduction2018-02-04bash-posh-1You're comfortable with PowerShell and you've honed your T-SQL skills over the years, but what about Bash?<p>You're comfortable with PowerShell and you've honed your T-SQL skills over the years, but what about the Bash scripting language? With the release of SQL on Linux a lot of DBAs are wondering how they might manage SQL on this new and unfamiliar operating system. While PowerShell Core is available and has come a long way, it still needs to be installed and certainly has some limitations. Bash on the other hand is available on almost any Linux system and can be incredibly powerful. </p>
<p>In this series, we'll cover what Bash actually is, some of the basics of using Bash, and my own opinions on how to best structure your scripts. My goal is to give people that are unfamiliar with Bash a good place to start. Throughout this post I will try to link to external documentation as much as possible, as the Bash <a href="http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html">documentation</a> is fantastic.</p>
<p>Also note that I will be trying to focus on writing scripts and less on individual commands. In this post we are going to start with writing a basic script to output information to the console.</p>
<h1>What is Bash?</h1>
<p>Bash (the <strong>B</strong>ourne <strong>A</strong>gain <strong>Sh</strong>ell) was created in 1989 for the GNU Project as a free replacement for the Unix Bourne shell. Most modern Linux systems use Bash as their default command line shell, so if you have ever dropped to a command line on a Linux system, you have probably used Bash. Just like PowerShell, Bash is both a scripting language and a command shell/interpreter. So not only can you execute commands in an interactive shell session, but you can also write scripts that incorporate multiple commands. </p>
<p>Once you get your hands dirty with Bash you'll notice a lot of features that were incorporated into PowerShell. Things like command substitution: <code>$(Get-Date)</code> were directly pulled from Bash <code>$(date)</code>. Other features will look familiar as well, like the ability to pipe multiple commands together.</p>
<p>One thing you need to understand right away is that Bash is string based, not object based like PowerShell. This means you'll find yourself doing a lot more string processing to get tasks done. Things like string splitting will be much more common. Bash does support objects, like arrays, but few if any commands output an array. As we go through this series you'll see that this might not be as limiting as it sounds.</p>
<h1>Setup</h1>
<p>Before we can get started writing Bash scripts we need to get a few things figured out. If you are not familiar with Linux and the command line you are going to need to get an environment set up and figure out which tools you will use to write your scripts.</p>
<h2>Test Environment</h2>
<p>Hopefully if you are here you have somewhere to run Bash scripts. If not, you have a lot of options:</p>
<ul>
<li><a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10">Windows Subsystem for Linux</a></li>
<li><a href="https://docs.docker.com/docker-for-windows/">Docker for Windows</a></li>
<li><a href="https://cloud.google.com/products/compute/">Google Compute</a></li>
</ul>
<p>There are plenty of cloud providers you could use to spin up a Linux VM, but I listed Google Compute here specifically because you can get a VM in their lowest performance tier for free. It's a great way to get started if you don't have access to a Linux system and don't want to make any changes to your system. All you need to get started is a Google account.</p>
<h2>Choice of Text Editor</h2>
<p>When it comes to your choice of text editor you have a lot of choices. By default most Linux distributions come with <code>vi</code>. There is a long -running war between users of <code>emacs</code> and <code>vi</code>. They both have their pros and cons so I am not going to tell you what to choose. I personally use <code>vi</code>, and you can find it on almost any Linux system. That being said, the learning curve is steep on both.</p>
<p>For those new to command line text editors I would suggest installing <code>nano</code> just to get started. Nano is available on most distributions and will be the closest thing to notepad.exe on the command line, documentation for using <code>nano</code> can be found <a href="https://www.nano-editor.org/dist/v2.9/nano.html">here</a>. Of course you can always use a graphical editor if you are working on a Linux desktop system or a Windows system with Bash installed.</p>
<h2>Using the Examples</h2>
<p>In all of our examples for the rest of this series, you'll need to copy and paste the script code directly into a file with the <code>.sh</code> extension, and execute it. The <code>.sh</code> extension isn't <em>required</em> but it makes finding script files in a directory listing a lot easier. Any time a command should be executed from a command line I will specifically call that out, and the example text will look like the following:</p>
<div class="codehilite"><pre><span></span><code>mark@atomo:~ >$ . ./myscript.sh
</code></pre></div>
<p>In the code above <code>mark@atomo:~ >$</code> is my command prompt, and <code>. ./myscript.sh</code> is the command we want to execute. The prompt shows the user and hostname, followed by the current directory. In this case our current directory is <code>~</code>, this is a shorthand for the users home directory.</p>
<h2>A Note About the Dot</h2>
<p>In Bash, the <code>.</code> has a few special purposes. The first is in filesystem navigation and interaction. In those contexts the <code>.</code> represents the current directory so, in the example above, <code>./myscript.sh</code> is referring to a script in our current directory. It's important to note here that Linux uses the forward slash (<code>/</code>) as a path separator, instead of the backslash (<code>\</code>) used in the Windows world.</p>
<p>Another purpose for the <code>.</code> is in script execution. The command above <em>starts</em> with a <code>.</code> followed by a space. In Bash there is a special command called <code>source</code>. This command takes in a file and executes it. The <code>.</code> you see that begins the command above is simply a shortcut, or "alias", to the <code>source</code> command. You could type: <code>source ./myscript.sh</code>, but using the <code>.</code> saves a few keystrokes. This is a way to execute script files that have not explicitly been set as executable.</p>
<h2>Your First Script</h2>
<p>No programming discussion is complete without the famous "Hello World" example. In this script we are introducing the "shebang" (<code>#!</code>), set options, and the <code>echo</code> command:</p>
<div class="codehilite"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="nb">set</span> -e
<span class="nb">set</span> -u
<span class="nb">echo</span> <span class="s2">"Hello World"</span>
</code></pre></div>
<p>Copy the following above code into a file called <code>hello_world.sh</code> in your home directory (<code>/home/<username></code> in Linux, <code>c:\users\<username></code> in Windows). If you are using nano you would type:</p>
<div class="codehilite"><pre><span></span><code>mark@atomo:~ >$ nano hello_world.sh
</code></pre></div>
<p>Then paste the code, save the file, and exit back to the command prompt.</p>
<h2>Explanation</h2>
<p>Before we execute our script, lets go over what we are doing in the script.</p>
<p>First, all Bash scripts should start with the same line:</p>
<div class="codehilite"><pre><span></span><code><span class="ch">#!/bin/bash</span>
</code></pre></div>
<p>The <code>#!</code> is known as a "shebang" in the Linux world, the path following the shebang specifies which interpreter should be used to execute the script. Since we are working with Bash we will set this to <code>/bin/bash</code>.</p>
<p>At this point you can start writing your script, but there are a few additional things I like to add first:</p>
<div class="codehilite"><pre><span></span><code><span class="nb">set</span> -e
<span class="nb">set</span> -u
</code></pre></div>
<p>These <code>set</code> options control the execution behavior of the script, and are optional, but I highly recommend using them. The <code>-e</code> option tells the script to fail on <em>any</em> instance where a command returns a non-zero exit code. This can make you script run in a more predictable manner. The <code>-u</code> option is useful for keeping your scripts clean. It will cause the script to fail if you declare a variable but never use it.</p>
<p>Finally we output a message to the screen using <code>echo</code>.</p>
<div class="codehilite"><pre><span></span><code><span class="nb">echo</span> <span class="s2">"Hello World"</span>
</code></pre></div>
<p>The <code>echo</code> command will simple output whatever text is passed to it, for the most part it resembles the PowerShell <code>Write-Host</code> cmdlet.
<code>echo</code> is the defacto way to output text to the console in most example you'll find on the internet, but in the next post in the series we are going to discuss a more flexible option called <code>printf</code>. <code>printf</code> has a few more features that make it worth using in place of <code>echo</code>.</p>
<h2>Executing the Script</h2>
<p>To execute this script you'll need to go to a Bash command prompt and type the following:</p>
<div class="codehilite"><pre><span></span><code>mark@atomo:~ >$ . ./hello_world.sh
</code></pre></div>
<p>And that's it! You should see "Hello World" echoed back to the console, and then you have been returned to a command prompt ready to execute somethign else.</p>
<h1>Final Thoughts</h1>
<p>Bash scripting can be a great skill to have when you start managing Linux systems. In this post we discussed options for setting up an environment to execute the example code, touched a little on the history of Bash, and ended with a simple example script to output "Hello World". In the next post of this series we are going to discuss console output using <code>echo</code> and <code>printf</code>, strings, and variables.</p>exciting news!2018-02-06news-01Announcing new projects and opportunities.<p>This isn't my usual post, but I wanted to tell people about a few cool projects I have been involved with recently that might interest some people. </p>
<h1>We Speak Linux</h1>
<p>I am super excited to be one of the people involved in this project early on. We Speak Linux is a place for Windows Administrators and SQL Server DBAs to learn more about the Linux operating system. There are plenty of good sources for information about Linux on the internet, but with WSL we wanted to build a community of people with a common goal: Transitioning their skills with the Windows operating system over to Linux. This group has really been driven by Tracy Boggiano (<a href="https://twitter.com/TracyBoggiano">T</a>|<a href="http://tracyboggiano.com">B</a>). She has been relentless in getting a Linux webinar-based training project off the ground, and has already scheduled a great starting line-up. Head over to our website to learn more: <a href="https://wespeaklinux.com">https://wespeaklinux.com/</a>.</p>
<h1>Speaking Mentors</h1>
<p>Another great project I am honored to be a part of is <a href="SpeakingMentors.com">http://speakingmentors.com</a>. I found out about this project from Alex Yates(<a href="https://twitter.com/_AlexYates_">T</a>|<a href="http://workingwithdevs.com">B</a>) on the SQL Community Slack and signed up to be a mentor right away. The concept is simple, but powerful. Speakers of any level of experience can come, choose a mentor from a list of seasoned presenters, and get help with anything they need. Need help working on your first talk? Need help honing your slide deck skills? Getting ready for your 200th talk? Whatever your needs are you can find help at <a href="http://speakingmentors.com">SpeakingMentors</a>. </p>
<h1>Telegraf</h1>
<p>With the reality of SQL on Linux soon approaching a workplace near me, I was trying to find a good metric collector that could run on both Windows and Linux to replace the Windows-only agent we were using. After some searching I found telegraf. Telegraf is an open-source cross-platform metric collection agent with a simple plugin framework that makes it very extensible. A SQL Server plugin existed but I didn't really agree with how, or how much, data was collected.</p>
<p>Being open-source I set to work rewritting the TSQL used by the plugin, and making a few small tweaks to the plugin itself. I am proud to say that my re-write has been accepted and my pull request was merged with the main project. From this point forward my revamped SQL Server plugin can be found in the nightly builds and will eventually be in the upcoming 1.6 release. Check the project out <a href="https://github.com/influxdata/telegraf">here</a>, and specifically the SQL Server plugin <a href="https://github.com/influxdata/telegraf/tree/master/plugins/inputs/sqlserver">here</a>. Be sure to keep an eye on Tracy Boggiano's <a href="http://tracyboggiano.com">blog</a> as she will be writing a little about our monitoring setup using Telegraf, InfluxDB, and Grafana.</p>bash output, variables, and strings2018-02-19bash-posh-2The basics of storing information in variables and controlling output to the console.<p>In the <a href="/bash-posh-1">last post</a> we talked a little about what Bash is and ended with a simple "Hello World" example. In this post we are going to keep adding to our script by adding a variable and exploring two options for handling console output.</p>
<h1>Console Output</h1>
<p>In the previous post we used <code>echo</code> to output a message to the console. A second option, which is a bit more flexible, is <code>printf</code>. <code>printf</code> supports a few additional features, like string formatting, that make it a better default choice for your output needs. Here is the script again with <code>printf</code>:</p>
<div class="codehilite"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="nb">set</span> -e
<span class="nb">set</span> -u
<span class="nb">printf</span> <span class="s2">"Hello World\n"</span>
</code></pre></div>
<p>One thing you might notice is the <code>\n</code> or "newline". Unlike <code>echo</code>, <code>printf</code> does not automatically insert a newline at the end of the line, it has to be added manually. I personally see this as a feature, as you get more control over the output of your text. This might not seem to be a good reason to use <code>printf</code>, and it isn't very exciting I know, but once we get into variables we can start to take advantage of more advanced features of <code>printf</code>.</p>
<h1>Variables</h1>
<p>If you have experience with PowerShell, some properties of Bash variables will feel familiar. In Bash, variables are denoted with a <code>$</code> just like in PowerShell, but unlike PowerShell the <code>$</code> is only needed when they are being referenced. When you are assigning a value to a variable, the <code>$</code> is left off:</p>
<div class="codehilite"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="nb">set</span> -e
<span class="nb">set</span> -u
<span class="nv">my_var</span><span class="o">=</span><span class="s2">"World"</span>
<span class="nb">printf</span> <span class="s2">"Hello </span><span class="si">${</span><span class="nv">my_var</span><span class="si">}</span><span class="s2">\n"</span>
</code></pre></div>
<p>Above we assigned a value to <code>my_var</code> without using the <code>$</code>, but when we then referenced it in the <code>printf</code> statement, we had to use a <code>$</code>. We also enclosed the variable name in curly braces. This is not required in all cases, but it is a good idea to get in the habit of using them. In cases where you are using positional parameters above 9 (we'll talk about this later) or you are using a variable in the middle of a string the braces are required, but there is no harm in adding them every time you use a variable in a string. Here is a quick example to illustrate:</p>
<div class="codehilite"><pre><span></span><code><span class="c1"># Braces required since there is no whitespace to the right of the variable</span>
<span class="nb">printf</span> <span class="s2">"mybig</span><span class="si">${</span><span class="nv">myvar</span><span class="si">}</span><span class="s2">string"</span>
<span class="c1"># Braces not required, there is whitespace to the right of the variable</span>
<span class="nb">printf</span> <span class="s2">"mybig</span><span class="nv">$myvar</span><span class="s2"> string"</span>
</code></pre></div>
<p>As I said, there is no harm in always using braces. It is not required in some cases, but it almost always makes your code more readable.</p>
<h1>Format String</h1>
<p>As mentioned above, <code>printf</code> has some more advanced features you can use when dealing with variables. While variable expansion is cool, you can also do the following with <code>printf</code>:</p>
<div class="codehilite"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="nb">set</span> -e
<span class="nb">set</span> -u
<span class="nv">my_var</span><span class="o">=</span><span class="s2">"World"</span>
<span class="nb">printf</span> <span class="s2">"Hello %s\n"</span> <span class="nv">$my_var</span>
</code></pre></div>
<p>This is a formatted string. In the example <code>%s</code> is replaced with the contents of the <code>$my_var</code> variable. String formatting is very powerful, in this example we are substituting a value into a string, but you can also do things like padding a string on the left or right, formatting a phone number, or controlling the precision of a floating point number. The ins and outs of formatting strings are best explained by the <a href="https://ss64.com/bash/printf.html">documentation</a>.</p>
<p>One interesting thing I will note here though is the special character <code>\a</code>. This allows you to "print" an audible alert. This can be useful if you want to get the users attention during a script execution. Give it a try your self, go to a Bash command prompt and type: <code>printf "\a"</code>. </p>
<h1>Strings</h1>
<p>This is a good time to go into strings a little deeper. Strings in Bash will be familiar for those with PowerShell experience. So far we have been enclosing our strings with double quotes, but you can also use single quotes. Like PowerShell, if you are trying to use variables directly in your strings (<code>"Hello ${my_var}"</code>) you will need to use double quotes. If you are using the format capabilities of <code>printf</code> you can use either. To illustrate, here is a script you can try:</p>
<div class="codehilite"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="nb">set</span> -e
<span class="nb">set</span> -u
<span class="nv">my_var</span><span class="o">=</span><span class="s2">"World"</span>
<span class="c1"># printf with single quotes. Variable expansion doesn't happen</span>
<span class="nb">printf</span> <span class="s1">'Hello ${my_var}\n'</span> <span class="c1"># Outputs 'Hello ${my_var}'</span>
<span class="c1"># printf with double quotes and variable expansion</span>
<span class="nb">printf</span> <span class="s2">"Hello </span><span class="si">${</span><span class="nv">my_var</span><span class="si">}</span><span class="s2">\n"</span> <span class="c1"># Outputs 'Hello World'</span>
<span class="c1"># printf with single quotes and a format string</span>
<span class="nb">printf</span> <span class="s1">'Hello %s\n'</span> <span class="nv">$my_var</span> <span class="c1"># Outputs 'Hello World'</span>
<span class="c1"># printf with double quotes and a format string</span>
<span class="nb">printf</span> <span class="s2">"Hello %s\n"</span> <span class="nv">$my_var</span> <span class="c1"># Outputs 'Hello World'</span>
</code></pre></div>
<p>In cases where you need to include quotes within your strings, you have a few options:</p>
<ul>
<li>If you need single quotes within your string, enclose the string in double quotes</li>
<li>If you need double quotes within your string and are not using any variables, enclose the string in single quotes</li>
<li>If you need double quotes within your string and are using variabels, you must escape the double quotes with a <code>\</code> </li>
</ul>
<p>Here are a few examples to illustrate:</p>
<div class="codehilite"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="nb">set</span> -e
<span class="nb">set</span> -u
<span class="nv">my_var</span><span class="o">=</span><span class="s2">"World"</span>
<span class="c1"># Double quotes nested in single quotes</span>
<span class="nb">printf</span> <span class="s1">'Hello "World"\n'</span>
<span class="c1"># Single quotes nested in double quotes</span>
<span class="nb">printf</span> <span class="s2">"Hello 'World'\n"</span>
<span class="c1"># Escaped double quotes nested in double quotes</span>
<span class="nb">printf</span> <span class="s2">"Hello \"World\"\n"</span>
<span class="c1"># Escaped double quotes nested in doulbe quotes with variable expansion</span>
<span class="nb">printf</span> <span class="s2">"Hello \"</span><span class="si">${</span><span class="nv">my_var</span><span class="si">}</span><span class="s2">\"\n"</span>
</code></pre></div>
<h1>Color!</h1>
<p>Now that we know how to print text to a console, and we know how to use variables, lets add a little pizzazz to our <code>Hello World</code> script:</p>
<div class="codehilite"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="nb">set</span> -e
<span class="nb">set</span> -u
<span class="c1"># Colors</span>
<span class="nv">red</span><span class="o">=</span><span class="s2">"`tput setaf 1`"</span>
<span class="nv">end</span><span class="o">=</span><span class="s2">"`tput sgr0`"</span>
<span class="nb">printf</span> <span class="s2">"</span><span class="si">${</span><span class="nv">red</span><span class="si">}</span><span class="s2">Hello</span><span class="si">${</span><span class="nv">end</span><span class="si">}</span><span class="s2"> World\n"</span>
</code></pre></div>
<p>Variables can be used for a lot of things, in this case we are storing some ANSI escape codes and using them to color our output. The codes are being generated by a small app called <code>tput</code>, you can read about using it (including a handy color table) <a href="https://ss64.com/bash/tput.html">here</a>. This can be useful when you are trying to make console output more clear, like making errors red for example.</p>
<h1>Final Thoughts</h1>
<p>User feedback is an important part of any script and you have a lot of options with Bash. There is even more to know about console output, but we are going to hold off on that until a later post when we cover error handling. Until then stay tuned for the next post in this series where we will discuss arithmetic operations, conditional logic, and looping.</p>my blog setup2018-03-07blog-setupHow to run a blog for less than $2 a month.<p>When you get a group of bloggers together it's inevitable that they'll start talking about the software they use to run their blog. Recent conversations about this very topic have inspired me to make a write up of the various components that go into my blog.</p>
<p>In this post I am going to discuss the platform I use, my workflow when publishing new posts, and how I can get TLS (https://), traffic analytics, a custom domain, search, and hosting on one of the worlds largest providers, all for under $2 a month. I'll go into detail on the more interesting bits, and link to other resources for the more standard setup pieces.</p>
<h1>Price List</h1>
<p>Here is the breakdown of all the tools I use and what the costs are (in USD):</p>
<table>
<thead>
<tr>
<th align="left">Tool</th>
<th align="left">Purpose</th>
<th align="right">Price</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">Google Domains</td>
<td align="left">Domain Registrar</td>
<td align="right">$12.00/yr ($1/month)</td>
</tr>
<tr>
<td align="left"><a href="#misc">Route 53</a></td>
<td align="left">DNS</td>
<td align="right">$0.50/month</td>
</tr>
<tr>
<td align="left"><a href="#aws-s3">S3</a></td>
<td align="left">Blog Hosting</td>
<td align="right">$0.14/month</td>
</tr>
<tr>
<td align="left"><a href="#cloudfront-cdn">Cloudfront</a></td>
<td align="left">CDN/TLS Proxy</td>
<td align="right">$0.13/month</td>
</tr>
<tr>
<td align="left"><a href="#misc">Mailgun</a></td>
<td align="left">Customer Domain Email</td>
<td align="right">FREE</td>
</tr>
<tr>
<td align="left"><a href="#git">Git Server</a></td>
<td align="left">Source Control</td>
<td align="right">FREE</td>
</tr>
<tr>
<td align="left">Google Analytics</td>
<td align="left">Traffic Analysis</td>
<td align="right">FREE</td>
</tr>
<tr>
<td align="left"><a href="#github-pages-and-beyond">Jekyll</a></td>
<td align="left">Static Site Generator</td>
<td align="right">FREE</td>
</tr>
<tr>
<td align="left"><a href="#search">Lunr.js</a></td>
<td align="left">In-Site Search</td>
<td align="right">FREE</td>
</tr>
<tr>
<td align="left"><a href="#tls-for-great-privacy">Let's Encrypt</a></td>
<td align="left">TLS Certificate</td>
<td align="right">FREE</td>
</tr>
<tr>
<td align="left"><a href="#jenkins">Jenkins</a></td>
<td align="left">Deploying static content</td>
<td align="right">FREE</td>
</tr>
<tr>
<td align="left">TOTAL</td>
<td align="left"></td>
<td align="right">$1.77/month</td>
</tr>
</tbody>
</table>
<p>If you can't tell from the list, I am a bit of a nerd. I love trying new technologies. Sure this would be easier to publish on something like Wordpress, but I love the amount of control I have over every part of my blog.</p>
<h1>GitHub Pages and Beyond</h1>
<p>My blog started it's life as a page on GitHub Pages. <a href="https://pages.github.com/">GitHub Pages</a> is a neat tool that serves a website from your github repository using a Ruby program called <a href="https://jekyllrb.com/">Jekyll</a>. Jekyll is a static site generator that uses a templating engine called <a href="https://shopify.github.io/liquid/">Liquid</a>. Static site generators are tools that programatically generate a website consisting of static HTML documents that can then be uploaded to practically any web host. This is opposed to a technology like PHP that generates content based on user requests and input. </p>
<p>Static site generators are nice because your blog posts are highly portable. Posts are typically written in a plain-text markup language like <a href="https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet">Markdown</a> so you can use any text editor to write posts. Here is a quick example of Markdown:</p>
<div class="codehilite"><pre><span></span><code># This is a heading
Here is some intro text
## Here is a sub-heading
* Bulleted list item 1
* Item 2
* Item 3
</code></pre></div>
<p>When I decided to move my blog from Github pages to my own servers I simply installed Jekyll on my local computer, generated my site, and uploaded the freshly generated website to my own server. Later when I moved to my current host (an S3 bucket in AWS), all I had to do was upload my website.</p>
<h1>AWS S3</h1>
<p><a href="https://aws.amazon.com/s3/">S3</a> is a cheap cloud storage solution provided by Amazon as part of their Amazon Web Services offering. S3 storage is presented as a "bucket" that you can store almost anything in. In my case I have a publicly readable bucket configured for static website hosting. AWS has some <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html">great documentation</a> on setting this up, it really only takes a few clicks of your mouse to get a static hosting bucket configured.</p>
<p>S3 pricing is based on the amount of data you are storing, and how many GET/PUT requests are made. While that might sound like it could get expensive, the pricing at the time I wrote this post is as follows:</p>
<p>|Storage - First 50TB/month|$0.0023/GB|
|PUT,LIST Requests|$0.005 per 1,000 requests|
|GET Requests|$0.0004 per 1,000 requests|</p>
<p>The great thing about hosting from S3 is the reliability. For pennies a month I get to host my small blog on servers that are managed by some of the most talented engineers in the world, on highliy redundant systems (your data is copied to at least three different physical facilities within a region).</p>
<h1>Cloudfront CDN</h1>
<p><a href="https://aws.amazon.com/cloudfront/">Cloudfront</a> is a CDN, or Content Distribution Network, run by Amazon. Cloudfront is responsible for caching static elements of websites and delivering those elements to a users web browser from one of hundreds of edge locations around the globe. My S3 bucket is located in the US-East region (in northern Virginia), with Cloudfront my site gets served from the edge location closest to the web client, this ensures the site always loads fast, no matter where you are accessing it from.</p>
<p>Cloudfront is also what allows me to use HTTPS. The certificate I get from Let's Encrypt is uploaded to Cloudfront and Cloudfront then uses that certificate to provide HTTPS connections to any clients trying to access my blog.</p>
<p>The process of setting up the S3 bucket, getting a certificate from Let's Encrypt, and setting up a Cloudfront distribution (this is what the Cloudfont instance is called) is actually pretty simple, instead of re-inventing the wheel though, I am just going to point you to this great tutorial: <a href="https://www.codeword.xyz/2016/01/06/lets-encrypt-a-static-site-on-amazon-s3/">Let's Encrypt a Static Site on Amazon S3</a>.</p>
<h1>TLS, for Great Privacy</h1>
<p>I am a huge fan of the current trend on the internet of sites adopting TLS (https://) over plain http. I am not going to go into my reasons because I think another site captures it quite well: <a href="https://doesmysiteneedhttps.com/">https://doesmysiteneedhttps.com/</a>.</p>
<p>When I first looked into getting a TLS certificate I went with a company called StartCom. Certificates were free but the process wasn't very simple, and couldn't be automated. I used StartCom for about a year before <a href="https://letsencrypt.org/">Let's Encrypt</a> started offering certificates. Let's Encrypt is an amazing free service that offers TLS certificates via API. With a simple shell script and a scheduled task you can automate the task of creating and renewing your certificate. I am currently using a pre-built docker container for this, check it out <a href="https://github.com/brycefisher/certbot-s3front">here</a>. </p>
<p>The combination of Let's Encrypt and docker has really simplified the process and makes the choice to go with https a no-brainer. With a single command I can generate a certificate and associate it with my Cloudfront distribution.</p>
<h1>Search</h1>
<h2>lunr.js</h2>
<p>Search was a fun one to implement. I had simple yet strict requirements: It must be fast, it must all run on the client side. After some digging I found the <a href="https://lunrjs.com/">lunr.js</a> project. Lunr is a javascript fulltext search engine. It supports all kinds of great features like boosting and word distance searches. The best thing about lunr is that it lives in a single blob of javascript and runs completely on the client.</p>
<h2>Search Index</h2>
<p>The easier thing to mess up when using lunr is the indexing process. Indexing your blog means you have to build an index that includes every post on your blog. If you aren't careful this index could be quite large and take a while to load to the browser. My approach involved writing a small custom Jekyll plugin in Ruby, and writing a minimal amount of information to the index.</p>
<p>The plugin is straight forward:</p>
<div class="codehilite"><pre><span></span><code><span class="k">module</span> <span class="nn">Jekyll</span>
<span class="k">module</span> <span class="nn">LunrStrip</span>
<span class="k">def</span> <span class="nf">lunr_strip</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
<span class="n">text</span><span class="o">.</span><span class="n">gsub</span><span class="p">(</span><span class="sr">/[^0-9a-z]/i</span><span class="p">,</span> <span class="s1">' '</span><span class="p">)</span><span class="o">.</span><span class="n">downcase</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">eat_small</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="n">min_len</span><span class="p">)</span>
<span class="n">text</span><span class="o">.</span><span class="n">gsub</span><span class="p">(</span><span class="sr">/\b.{1,</span><span class="si">#{</span><span class="n">min_len</span><span class="si">}</span><span class="sr">}\b/i</span><span class="p">,</span> <span class="s1">' '</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="no">Liquid</span><span class="o">::</span><span class="no">Template</span><span class="o">.</span><span class="n">register_filter</span><span class="p">(</span><span class="no">Jekyll</span><span class="o">::</span><span class="no">LunrStrip</span><span class="p">)</span>
</code></pre></div>
<p>I saved this file as in the <code>_plugins</code> directory under the root of my blog. Jekyll automatically looks here for plugins when building your blog. I used this plugin to programatically generate my search index every time I build my blog. Whenever my blog is built, the index is built and stored in this file: <a href="/js/search.js">/js/search.js</a>. The heavy lifting is accomplished using the following liquid template snippet:</p>
<div class="codehilite"><pre><span></span><code><span class="p">{{</span><span class="s2">"{%"</span><span class="p">}}</span> assign count = 0 %}<span class="p">{{</span><span class="s2">"{%"</span><span class="p">}}</span> for post in site.posts %}
this.add({
title: <span class="p">{{</span><span class="s2">"{{"</span><span class="p">}}</span>post.title | jsonify}},
tags: <span class="p">{{</span><span class="s2">"{{"</span><span class="p">}}</span>post.category | jsonify}},
content: <span class="p">{{</span><span class="s2">"{{"</span><span class="p">}}</span>post.content | lunr_strip | eat_small: 3 | split: " " | sort | uniq | join: " " | jsonify}},
summary: <span class="p">{{</span><span class="s2">"{{"</span><span class="p">}}</span>post.summary | strip_html | jsonify}},
id: <span class="p">{{</span><span class="s2">"{{"</span><span class="p">}}</span>count}}
});<span class="p">{{</span><span class="s2">"{%"</span><span class="p">}}</span> assign count = count | plus: 1 %}<span class="p">{{</span><span class="s2">"{%"</span><span class="p">}}</span> endfor %}
})
</code></pre></div>
<p>The magic happens in the <code>content</code> line. There we take the entire contents of the post, pass it through the custom Jekyll plugin to strip out all non alphanumeric characters and remove words with less than three characters in them. After that we convert the remaining text into an array, sort the array, and remove duplicate entries. We then join the array back into a single string and convert it to a JSON compliant string and add it to the search index located in the <code>search.js</code> file starting around line 10.</p>
<p>This whole process leaves us with a list of unique words for each post. This makes the file size much smaller while still allowing us to search accurately. As of this writing the search index for my entire blog weighs in at less than 60kb. While this search is working great I would love to remove the need for the client to download the search index. In the future I may try to use a lambda function that performs all of the search operations on the users behalf.</p>
<h1>Workflow</h1>
<p>My current workflow consists of whatever text editor I feel like using, git, and Jenkins. Jenkins is a very recent addition to my workflow. Previously I would write posts, build my blog, then run a script to upload it to S3. My biggest challenge with that setup was that I had to be on my laptop to deploy my blog. If I wanted to make a quick correction from my phone (I currently use the Tig Git client on my iPhone to write posts) I would have to check in my changes and then hop on my laptop to deploy the changes. Then I discovered Jenkins.</p>
<h2>Jenkins</h2>
<p>I had heard about Jenkins before but I had never really seen it in action until recently when I attended <a href="http://opensource101.com/raleigh/">Open Source 101</a>, a local open source software conference. There I attended a <a href="http://opensource101.com/raleigh/talks/jenkins-2-coding-continous-delivery-pipelines/">Jenkins session</a> by Brent Laster. This session was the first time I had been exposed to Jenkins pipeline jobs. </p>
<p>Using Jenkins pipeline jobs you can define the various steps in your build process using a custom version of the <a href="http://www.groovy-lang.org/">groovy</a> scripting language. This code is then checked in right along side your blog. After that it is as simple as pointing Jenkins at your blog repo. Here is the <code>Jenkinsfile</code> that defines my build process:</p>
<div class="codehilite"><pre><span></span><code><span class="n">pipeline</span> <span class="o">{</span>
<span class="n">agent</span> <span class="n">any</span>
<span class="n">stages</span> <span class="o">{</span>
<span class="n">stage</span><span class="o">(</span><span class="s1">'Build'</span><span class="o">)</span> <span class="o">{</span>
<span class="n">steps</span> <span class="o">{</span>
<span class="n">echo</span> <span class="s1">'Building..'</span>
<span class="n">sh</span><span class="o">(</span><span class="s1">'PATH="/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin"; bundle install; bundle exec jekyll build --incremental'</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="n">stage</span><span class="o">(</span><span class="s1">'Prepare for Deployment'</span><span class="o">)</span> <span class="o">{</span>
<span class="n">steps</span> <span class="o">{</span>
<span class="n">echo</span> <span class="s1">'Minifying Style Sheets...'</span>
<span class="n">sh</span><span class="o">(</span><span class="s1">'_deploy/minify.sh _site/public/css/print.css'</span><span class="o">)</span>
<span class="n">sh</span><span class="o">(</span><span class="s1">'_deploy/minify.sh _site/public/css/style.css'</span><span class="o">)</span>
<span class="n">sh</span><span class="o">(</span><span class="s1">'_deploy/minify.sh _site/public/css/syntax.css'</span><span class="o">)</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="n">stage</span><span class="o">(</span><span class="s1">'Deploy'</span><span class="o">)</span> <span class="o">{</span>
<span class="n">steps</span> <span class="o">{</span>
<span class="n">echo</span> <span class="s1">'Deploying....'</span>
<span class="n">withCredentials</span><span class="o">([[</span><span class="n">$class</span><span class="o">:</span> <span class="s1">'UsernamePasswordMultiBinding'</span><span class="o">,</span> <span class="nl">credentialsId:</span> <span class="s1">'blog-s3'</span><span class="o">,</span><span class="nl">usernameVariable:</span> <span class="s1">'USERNAME'</span><span class="o">,</span> <span class="nl">passwordVariable:</span> <span class="s1">'PASSWORD'</span><span class="o">]])</span> <span class="o">{</span>
<span class="n">sh</span> <span class="s1">'_deploy/deploy_to_s3.sh $USERNAME $PASSWORD'</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div>
<p>This code is fairly straight forward. First it makes sure all the proper dependencies are installed for Jekyll, and builds the site. Then I run my CSS assets through a custom minify script I wrote. This makes the files smaller, which decreases load time and bandwidth usage. Finally I retrieve my AWS credentials and run a custom deploy script to copy all of my blog files over to my S3 bucket.</p>
<p>You can read more about writing pipeline jobs via a Jenkinsfile <a href="https://jenkins.io/doc/book/pipeline/jenkinsfile/">here</a>.</p>
<h2>Git</h2>
<p>There isn't much special about my Git setup expect one thing. Jenkins has a feature that allows you to trigger a build via a simple web request to the Jenkins server. To accomplish this I added the following to the a <code>post-update</code> script in the <code>hooks</code> directory on my git server. The script simply makes a web request whenever code is pushed to the master branch of my blog project:</p>
<div class="codehilite"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="nv">branch</span><span class="o">=</span><span class="k">$(</span>git rev-parse --symbolic --abbrev-ref <span class="nv">$1</span><span class="k">)</span>
<span class="nb">echo</span> <span class="s2">"Commit received for branch: </span><span class="si">${</span><span class="nv">branch</span><span class="si">}</span><span class="s2">"</span>
<span class="k">if</span> <span class="o">[[</span> <span class="s2">"</span><span class="si">${</span><span class="nv">branch</span><span class="si">}</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"master"</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
<span class="nb">echo</span> <span class="s2">"Calling build API..."</span>
curl -k https://<api-user>:<api-user-key>@<jenkins-server-address>/job/Blog/build?token<span class="o">=</span><auth-token>
<span class="k">fi</span>
<span class="nb">exec</span> git update-server-info
</code></pre></div>
<h1>Misc</h1>
<p>There are a few things I didn't really cover in detail, but wanted to call out here:</p>
<ul>
<li><a href="https://simplyian.com/2015/01/07/Hacking-GMail-to-use-custom-domains-for-free/">Hacking Gmail to use custom domains for free</a> - using Mailgun in combination with Gmail for custom domain email for free</li>
<li><a href="https://michaelsoolee.com/google-analytics-jekyll/">Google Analytics setup for Jekyll</a></li>
<li><a href="https://git-scm.com/book/en/v2/Git-on-the-Server-Setting-Up-the-Server">Setting up a Git Server</a></li>
<li><a href="https://aws.amazon.com/route53/">Route 53</a> - this is Amazons cloud DNS service, at the time I set this up, it was required when using Cloudfront and TLS, this may no longer be the case.</li>
</ul>
<h1>Final Thoughts</h1>
<p>This whole mess of components looks really complex when I type it all up, but it has been an ongoing project that has grown as I find new technologies to use and new challenges. I think of all the projects I have worked on in the past, setting up my blog is the one that really illustrates how I learn about technology. Before this setup I had redundant web servers with a load balancer running HAPROXY, mostly just because I wanted to learn how to use HAPROXY.</p>
<p>The amount of free/cheap technologies out there is truly astounding, and none of this would be possible without open source software, and people curious enough and driven enough to create these wonderful things. While I know this setup isn't for everyone, I hope someone finds some of this useful. or it inspires someone to try hosting their blog on their own.</p>bash arithmetic and conditional logic2018-03-21bash-posh-3How to handle basic integer and floating point arithmetic in Bash, and how to add conditional logic to your scripts.<p>Expanding on the hello world script we have been looking at in <a href="/series/#bash-for-the-powershell-developer">previous posts</a>, we will add some intelligence to our greeting using conditional logic. We'll also take a look at a few different methods for executing common arithmetic operations.</p>
<blockquote>
<p>When I originally started writing this post I intended on including a section on looping, due to the length of this post I decided to leave it off and include it in the next installment of this series.</p>
</blockquote>
<h1>Good Morning World</h1>
<p>Last we looked at our hello world script we had managed to add some color and use a variable to store part of our greeting. While that wasn't very exciting it lays the foundation to do some more interesting things.</p>
<p>In this example we are going to say <code>Good Morning World</code>, <code>Good Afternoon World</code>, and <code>Good Evening World</code> depending on the time of day. Let's take a look at the script and then walk through what is happening.</p>
<div class="codehilite"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="nb">set</span> -e
<span class="nb">set</span> -u
<span class="nv">hour_of_day</span><span class="o">=</span><span class="k">$(</span>date +%H<span class="k">)</span>
<span class="k">if</span> <span class="o">[[</span> <span class="nv">$hour_of_day</span> -lt <span class="m">12</span> <span class="o">&&</span> <span class="nv">$hour_of_day</span> -ge <span class="m">5</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
<span class="nv">time_of_day</span><span class="o">=</span><span class="s2">"Morning"</span>
<span class="k">elif</span> <span class="o">[</span> <span class="nv">$hour_of_day</span> -lt <span class="m">18</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
<span class="nv">time_of_day</span><span class="o">=</span><span class="s2">"Afternoon"</span>
<span class="k">else</span>
<span class="nv">time_of_day</span><span class="o">=</span><span class="s2">"Evening"</span>
<span class="k">fi</span>
<span class="nb">printf</span> <span class="s2">"Good </span><span class="si">${</span><span class="nv">time_of_day</span><span class="si">}</span><span class="s2"> World\n"</span>
</code></pre></div>
<h1>Command Substitution</h1>
<p>The first interesting thing about the new additions to our script is the <code>$(date +%H)</code>. This is referred to as command substitution. Command substitution executes whatever command is specified within the <code>$(</code> <code>)</code> and treats the result as a string. In this example it is returning the hour of the day and storing that in the <code>$hour_of_day</code> variable.</p>
<h1>Conditional Logic</h1>
<p>Next we move on to the conditional logic. In this case we are using the <code>if</code> command. The basic structure of an if command is as follows:</p>
<div class="codehilite"><pre><span></span><code><span class="k">if</span> <span class="o">[</span> <some expression> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
<commands>
<span class="k">fi</span>
</code></pre></div>
<p>One thing to remember is that if you want to string together several conditions in a single expression, you have to use the <code>&&</code> (and) or the <code>||</code> (or) operators. In our example we are using <code>&&</code>. Whenever you do this you also need to enclose the expression in a double set of square brackets:</p>
<div class="codehilite"><pre><span></span><code><span class="k">if</span> <span class="o">[[</span> <some expression> <span class="o">&&</span> <some other expression> <span class="o">]]</span><span class="p">;</span> <span class="k">then</span>
<commands>
<span class="k">fi</span>
</code></pre></div>
<h2>Expressions</h2>
<p>An expression is a statement that evaluates to either true or false. In our case we are just checking if the <code>$hour_of_day</code> variable is less than 12 and greater than or equal to 5. When comparing numeric values you must use the same operators you are familiar with from PowerShell: <code>-lt</code> <code>-gt</code> <code>-le</code> <code>-ge</code> <code>-ne</code>. When comparing string values you have to use operators you might be more used to from other languages: <code>==</code> <code>!=</code> and both values must be enclosed in double quotes <code>[ "string1" == "string2" ]</code>.</p>
<h2>Else If</h2>
<p>The <code>if</code> command also has support for <code>else</code> and <code>elif</code> (else if), to allow you to set up more complex logical checks. The structure doesn't change much but there a few things to call out: </p>
<div class="codehilite"><pre><span></span><code><span class="k">if</span> <span class="o">[</span> <some expression> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
<commands>
<span class="k">elif</span> <span class="o">[</span> <some expression> <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
<commands>
<span class="k">else</span>
<commands>
<span class="k">fi</span>
</code></pre></div>
<p>Looking at the example above, <code>if</code> and <code>elif</code> behave roughly the same. They expect an expression and a command terminator <code>;</code>, and must be followed by a <code>then</code> statement. Where they differ is that the <code>elif</code> does not have it's own closing statement, it relies on the closing <code>fi</code> instead. The <code>else</code> statement is even simpler, not requiring an expression or a <code>then</code> statement. </p>
<p>In our example we are using the <code>elif</code> to check if the <code>$hour_of_day</code> variable is less than 18 but greater than 12, and finally the <code>else</code> to catch cases where the <code>$hour_of_day</code> is greater than 18.</p>
<p>There are a huge number of expressions you can use in your <code>if</code> command (including regex matching). A deep dive into <code>if</code> is outside the scope of this post, but there is a great guide on all the options here: <a href="http://www.tldp.org/LDP/Bash-Beginners-Guide/html/sect_07_01.html">Introduction to If</a>.</p>
<h1>A Case for CASE</h1>
<p>While the <code>if</code>/<code>elif</code>/<code>else</code> construct is nice an familiar, sometimes a simple <code>case</code> statement will do the trick. Here is an example of the <code>if</code> logic from our first example using a case statement:</p>
<div class="codehilite"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="nb">set</span> -e
<span class="nb">set</span> -u
<span class="nv">hour_of_day</span><span class="o">=</span><span class="k">$(</span>date +%H<span class="k">)</span>
<span class="k">case</span> <span class="nv">$hour_of_day</span> <span class="k">in</span>
<span class="o">[</span><span class="m">5</span>-11<span class="o">]</span>*<span class="o">)</span>
<span class="nv">time_of_day</span><span class="o">=</span><span class="s2">"Morning"</span>
<span class="p">;;</span>
<span class="o">[</span><span class="m">12</span>-17<span class="o">]</span>*<span class="o">)</span>
<span class="nv">time_of_day</span><span class="o">=</span><span class="s2">"Afternoon"</span>
<span class="p">;;</span>
*<span class="o">)</span>
<span class="nv">time_of_day</span><span class="o">=</span><span class="s2">"Evening"</span>
<span class="p">;;</span>
<span class="k">esac</span>
<span class="nb">printf</span> <span class="s2">"Good </span><span class="si">${</span><span class="nv">time_of_day</span><span class="si">}</span><span class="s2"> World\n"</span>
</code></pre></div>
<p>This code accomplishes the same thing as the <code>if</code> logic, it just looks a bit different. I recommed reading up on <code>case</code> as it can be useful, but as long as you have a good grasp of <code>if</code>/<code>elif</code>/<code>else</code> you should have everything you need in most cases. Here are a few examples of using <code>case</code> in different ways: <a href="https://www.thegeekstuff.com/2010/07/bash-case-statement/">5 Bash Case Statement Examples</a></p>
<h1>Arithmetic</h1>
<p>Doing arithmetic in bash is not nearly as straight forward as it is in PowerShell, but it is still possible.</p>
<h2>Integers</h2>
<p>Integer arithmetic can be accomplished with this simple syntax:</p>
<div class="codehilite"><pre><span></span><code><span class="o">((</span> <span class="nv">my_var</span> <span class="o">=</span> <span class="m">1</span> + <span class="m">2</span> <span class="o">))</span>
<span class="nb">printf</span> <span class="s2">"</span><span class="si">${</span><span class="nv">my_var</span><span class="si">}</span><span class="s2">\n"</span>
</code></pre></div>
<p>All we need to do for integer arithmetic is enclose the operation (and variable assignment) in double parenthesis. This works for all types of operations, but always be mindful that any decimals will be discarded. For example:</p>
<div class="codehilite"><pre><span></span><code><span class="o">((</span> <span class="nv">my_var</span> <span class="o">=</span> <span class="m">5</span> / <span class="m">2</span> <span class="o">))</span>
<span class="nb">printf</span> <span class="s2">"</span><span class="si">${</span><span class="nv">my_var</span><span class="si">}</span><span class="s2">\n"</span>
</code></pre></div>
<p>This command will print a <code>2</code> to the screen. The remaining <code>0.5</code> has been discarded. This method also respects order of operations, and grouping operations in additional parenthesis:</p>
<div class="codehilite"><pre><span></span><code><span class="o">((</span> <span class="nv">my_var</span> <span class="o">=</span> <span class="m">5</span> * <span class="o">(</span><span class="m">2</span> + <span class="m">1</span><span class="o">)</span> <span class="o">))</span>
<span class="nb">printf</span> <span class="s2">"</span><span class="si">${</span><span class="nv">my_var</span><span class="si">}</span><span class="s2">\n"</span>
</code></pre></div>
<h2>Floating Point Numbers</h2>
<p>Since there is no native support for floating point numbers in bash, we have to use an external utility called <code>bc</code> which is available on most systems (with the exception of some embedded systems):</p>
<div class="codehilite"><pre><span></span><code><span class="nb">printf</span> <span class="s2">"</span><span class="k">$(</span><span class="nb">echo</span> <span class="s2">"scale=2;5 / 2"</span> <span class="p">|</span> bc<span class="k">)</span><span class="s2">\n"</span>
</code></pre></div>
<p>This is not the prettiest I will admit. Using <code>bc</code> we have to "feed" it the equation and scale (precision) information by way of <code>echo</code> (or any other standard input method). If you do not supply a <code>scale</code> in your command, <code>bc</code> will behave differently than expected for some operations. For example, if you try: </p>
<div class="codehilite"><pre><span></span><code><span class="nb">printf</span> <span class="s2">"</span><span class="k">$(</span><span class="nb">echo</span> <span class="s2">"5 / 2"</span> <span class="p">|</span> bc<span class="k">)</span><span class="s2">\n"</span>
</code></pre></div>
<p>You will see it returns <code>2</code>. To avoid this, you can either set the <code>scale</code> with each command or create a configuration file.</p>
<h2>bc Configuration</h2>
<p>To create a configuration file, create a new file in your home directory called <code>.bc</code> and put the following in it:</p>
<div class="codehilite"><pre><span></span><code>scale=2
</code></pre></div>
<p>The scale can be whatever you like. Then you need to set an environment variable to tell bc where to find the config file. The best way to do this is to add the following to your <code>.bash_profile</code> file:</p>
<div class="codehilite"><pre><span></span><code>BC_ENV_ARGS='/home/<your user name here>/.bc'
export BC_ENV_ARGS
</code></pre></div>
<p>To immediately apply this change, just type the following at a command prompt:</p>
<div class="codehilite"><pre><span></span><code>mark@atomo:~ >$ <span class="nb">source</span> ~/.bash_profile
</code></pre></div>
<p><code>bc</code> can do almost any mathmatical operation you can think of. To read more about <code>bc</code> take a look at the manual over at gnu.org: <a href="https://www.gnu.org/software/bc/manual/html_mono/bc.html">bc Command Manual</a></p>
<h1>Final Thoughts</h1>
<p>Conditional logic and arithmetic are some of the most important building blocks when constructing more complex scripts. While native arithmetic operations are a bit limited in Bash, <code>bc</code> more than makes up for it. In the next post we are going to spend some time looking at looping methods using <code>for</code>, <code>while</code>, and <code>xargs</code>.</p>something new2020-01-06something_newA new blog with a new focus.<p>Hello friends, welcome to the new blog! My name is Mark Wilkinson and I live in the small town of Garner, outside of Raleigh North Carolina. I live here with my wife, four children, dog, and seven chickens. I've been working in technology for over 15 years in one capacity or another. In my current role I lead a group of outstanding DBAs at a company called ChannelAdvisor in Morrisville. </p>
<p>I started this blog to focus more on me and what is happening in my life and less on SQL Server (like I do here: <a href="https://m82labs.com">m82labs</a>). While I do enjoy SQL Server and you will see some SQL Server posts here, I'll primarily be focussing on whatever I currently find interesting with a smattering of commentary on articles/books I've read. Currently I do a lot of professional work in PowerShell and a little bit of Python. In my personal life I use a lot of Python and Bash, <a href="https://github.com/influxdata/telegraf/tree/master/plugins/inputs/sqlserver">some Go</a>, and a little PowerShell. I've also taken to working on craft projects with my wife (what a great reason to buy new tools). </p>
<p>This blog runs on a static site generator I wrote in Python3 called Antiquity (that I don't plan on releasing) that attempts to output mostly plain text, responsive web pages. I've been using static site generators for a while, starting on Github pages and eventually moving to a Jekyll blog running in an S3 bucket. I wanted to write my own just to see how hard it was, and to see how light weight I could make it. My current setup is very similar to many others:</p>
<ul>
<li>Uses markdown for the post format</li>
<li>Uses yaml frontmatter on each post to determine posting date, title, short link, and draft status</li>
<li>Supports post "collections"</li>
<li>Uses a custom templating system (just find/replace at this point)</li>
<li>Supports code formatting via pygments</li>
</ul>
<p>I hope you enjoy what you see here. I'm primarily trying to blog for myself with this blog, so you won't see any buttons to share my posts on social media, and you won't see anything trying to track you (if you do, <a href="/about/">PLEASE CONTACT ME!</a>)</p>powershell classes (without classes)2020-01-07posh-classes-withoutImplementing class functionality in older versions of PowerShell.<p>The "Classes" feature in PowerShell v5+ can be super useful, but what happens if you need class-like functionality but are forced to use an older version of PowerShell? Never fear! We can get much of the functionality we see when using classes using the <code>PSCustomObject</code> type.</p>
<p>This post was inspired by a conversation I recently had with a friend. He had implemented a module that relied on classes only to find out that he needed to support some systems running PowerShell 4. With an upgrade being out of the question, I came up with an example that could get him most of what he was looking for.</p>
<blockquote>
<p>If you haven't used PS classes before, I suggest you read up on them: <a href="https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_classes?view=powershell-6">Microsoft Docs: About Classes</a>. Classes are great when you are working on a project where a compiled language might be overkill but you need tighter control over data types, or need to pass complex objects around between functions. </p>
</blockquote>
<h2>PSCustomObject</h2>
<p><code>PSCustomObject</code> has been around since PowerShell v3.0. <code>PSCustomObject</code> is similar to a hashtable in that it can store named properties, but it behaves a bit differently when displaying the data (much nicer formatting). The killer feature though, is the ability to not only add properties but also methods (just like classes!) to your object.</p>
<p>Lets say we wanted a class the would expect a number when the object was created, and then allowed you to do some simple math on the number via class methods. I know this seems like a silly use case, but it very clearly shows how you can use custom objects as if they were classes. </p>
<p>If you were developing this in PowerShell v5+, you might use a class:</p>
<div class="codehilite"><pre><span></span><code><span class="n">class</span> <span class="n">MyClass</span> <span class="p">{</span>
<span class="no">[int]</span><span class="nv">$SomeNum</span>
<span class="n">MyClass</span><span class="p">(</span><span class="no">[int]</span><span class="nv">$SomeNum</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$this</span><span class="p">.</span><span class="n">SomeNum</span> <span class="p">=</span> <span class="nv">$SomeNum</span>
<span class="p">}</span>
<span class="no">[int]</span> <span class="n">TimesFive</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="nv">$this</span><span class="p">.</span><span class="n">SomeNum</span> <span class="p">*</span> <span class="n">5</span>
<span class="p">}</span>
<span class="no">[int]</span> <span class="n">TimesNum</span><span class="p">(</span><span class="no">[int]</span><span class="nv">$Num</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nv">$this</span><span class="p">.</span><span class="n">SomeNum</span> <span class="p">*</span> <span class="nv">$Num</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nv">$MyClassInstance</span> <span class="p">=</span> <span class="no">[MyClass]</span><span class="p">::</span><span class="n">New</span><span class="p">(</span><span class="n">5</span><span class="p">)</span>
<span class="nv">$MyClassInstance</span><span class="p">.</span><span class="n">TimesFive</span><span class="p">()</span>
</code></pre></div>
<p>This code creates a new class called <code>MyClass</code>, then we instantiate a new instance of the class with <code>$SomeNum</code> defined as <code>5</code>.</p>
<p>Now, let's do the same thing using <code>PSCustomObject</code> and <code>Add-Member</code>:</p>
<div class="codehilite"><pre><span></span><code><span class="k">function</span> <span class="nb">New-MyFakeClass</span> <span class="p">{</span>
<span class="k">param</span><span class="p">(</span>
<span class="nv">$SomeNum</span>
<span class="p">)</span>
<span class="nv">$MyFakeClass</span> <span class="p">=</span> <span class="no">[PSCustomObject]</span><span class="p">@{}</span>
<span class="no">[scriptblock]</span><span class="nv">$TimesFive</span> <span class="p">=</span> <span class="p">{</span>
<span class="cm"><#</span>
<span class="cm"> Returns "SomeNum" multiplied by five</span>
<span class="cm"> #></span>
<span class="nv">$this</span><span class="p">.</span><span class="n">SomeNumber</span> <span class="p">*</span> <span class="n">5</span>
<span class="p">}</span>
<span class="no">[scriptblock]</span><span class="nv">$TimesNum</span> <span class="p">=</span> <span class="p">{</span>
<span class="cm"><#</span>
<span class="cm"> Returns "SomeNum" multiplied by an arbitrary integer</span>
<span class="cm"> #></span>
<span class="k">param</span><span class="p">(</span>
<span class="no">[int]</span><span class="nv">$Num</span>
<span class="p">)</span>
<span class="nv">$this</span><span class="p">.</span><span class="n">SomeNumber</span> <span class="p">*</span> <span class="nv">$Num</span>
<span class="p">}</span>
<span class="nv">$MyFakeClass</span> <span class="p">|</span> <span class="nb">Add-Member</span> <span class="n">-MemberType</span> <span class="n">NoteProperty</span> <span class="n">-Name</span> <span class="n">SomeNumber</span> <span class="n">-Value</span> <span class="nv">$SomeNum</span>
<span class="nv">$MyFakeClass</span> <span class="p">|</span> <span class="nb">Add-Member</span> <span class="n">-MemberType</span> <span class="n">ScriptMethod</span> <span class="n">-Name</span> <span class="n">TimesFive</span> <span class="n">-Value</span> <span class="nv">$TimesFive</span>
<span class="nv">$MyFakeClass</span> <span class="p">|</span> <span class="nb">Add-Member</span> <span class="n">-MemberType</span> <span class="n">ScriptMethod</span> <span class="n">-Name</span> <span class="n">TimesNum</span> <span class="n">-Value</span> <span class="nv">$TimesNum</span>
<span class="nv">$MyFakeClass</span>
<span class="p">}</span>
<span class="nv">$MyFakeClassInstance</span> <span class="p">=</span> <span class="nb">New-MyFakeClass</span> <span class="n">-SomeNum</span> <span class="n">5</span>
<span class="nv">$MyFakeClassInstance</span><span class="p">.</span><span class="n">TimesFive</span><span class="p">()</span>
</code></pre></div>
<p>This is a bit more code, so lets break it down a little. </p>
<p>First we create a "factory" function <code>New-MyFakeClass</code> (a function which creates new objects):</p>
<div class="codehilite"><pre><span></span><code><span class="k">function</span> <span class="nb">New-MyFakeClass</span> <span class="p">{</span>
<span class="k">param</span><span class="p">(</span>
<span class="nv">$SomeNum</span>
<span class="p">)</span>
</code></pre></div>
<p>Since we aren't using classes, we have to create a custom function to take the place of the class constructor. In a class, the constructor is responsible for initializing a new class object (setting initial values for properties, etc.).</p>
<p>Within this function we create an empty object called $MyFakeClass. Next we define two script blocks. These script blocks take the place of the method definitions in the class-based example above. Once we have those defined we can start adding members to the object:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$MyFakeClass</span> <span class="p">|</span> <span class="nb">Add-Member</span> <span class="n">-MemberType</span> <span class="n">NoteProperty</span> <span class="n">-Name</span> <span class="n">SomeNumber</span> <span class="n">-Value</span> <span class="nv">$SomeNum</span>
<span class="nv">$MyFakeClass</span> <span class="p">|</span> <span class="nb">Add-Member</span> <span class="n">-MemberType</span> <span class="n">ScriptMethod</span> <span class="n">-Name</span> <span class="n">TimesFive</span> <span class="n">-Value</span> <span class="nv">$TimesFive</span>
<span class="nv">$MyFakeClass</span> <span class="p">|</span> <span class="nb">Add-Member</span> <span class="n">-MemberType</span> <span class="n">ScriptMethod</span> <span class="n">-Name</span> <span class="n">TimesNum</span> <span class="n">-Value</span> <span class="nv">$TimesNum</span>
</code></pre></div>
<p>This adds a property named <code>SomeNumber</code> of type <code>NoteProperty</code>. <code>NoteProperty</code> members act just as a class property would, you can easily access them via: <code>$MyFakeClassInstance.SomeNumber</code>. After that we now add our two script blocks, but this time we use a type of <code>ScriptMethod</code>. The interesting thing about <code>ScriptMethod</code> is that the script blocks get access to the <code>$this</code> variable just like in a more formal class object. This gives the script methods access to the internal properties of the class instance, in this case the <code>SomeNumber</code> property.</p>
<p>Finally we can see how we use this new function to create new objects, and how we can access the <code>ScriptMethod</code> just like a standard class method: </p>
<div class="codehilite"><pre><span></span><code><span class="nv">$MyFakeClassInstance</span> <span class="p">=</span> <span class="nb">New-MyFakeClass</span> <span class="n">-SomeNum</span> <span class="n">5</span>
<span class="nv">$MyFakeClassInstance</span><span class="p">.</span><span class="n">TimesFive</span><span class="p">()</span>
</code></pre></div>
<h2>Thoughts</h2>
<p>Another friend of mine told me that when you start using classes in PowerShell, it's time to look at a compiled language. In some cases I agree, but the maintainability of PowerShell when your team is NOT made up of software developers just can't be beat. I highly suggest reading about both <code>Class</code> and <code>PSCustomObject</code>. Even on PowerShell v5+, BOTH can be used to great effect.</p>runspaces explained2020-02-06runspaces-explainedGetting started with runspaces<p>PowerShell runspaces are a great, if often confusing, feature of PowerShell. If you need to get a lot of work done fast, and have capacity to do lots of work in parallel, runspaces can help you out.</p>
<p>In this series of posts on runspaces I hope to give you the information you need to understand, use, and troubleshoot runspaces more effectively.</p>
<h2>simple runspaces example</h2>
<p>Below is the typical code you might see when reading about working with runspaces:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$RunspacePool</span> <span class="p">=</span> <span class="no">[RunspaceFactory]</span><span class="p">::</span><span class="n">CreateRunspacePool</span><span class="p">(</span><span class="n">1</span><span class="p">,</span> <span class="n">5</span><span class="p">)</span>
<span class="nv">$RunspacePool</span><span class="p">.</span><span class="n">Open</span><span class="p">()</span>
<span class="nv">$ScriptBlock</span> <span class="p">=</span> <span class="p">{</span> <span class="nb">Get-Random</span> <span class="p">}</span>
<span class="nv">$Runspaces</span> <span class="p">=</span> <span class="p">@()</span>
<span class="p">(</span><span class="n">1</span><span class="p">..</span><span class="n">10</span><span class="p">)</span> <span class="p">|</span> <span class="k">ForEach</span><span class="n">-Object</span> <span class="p">{</span>
<span class="nv">$Runspace</span> <span class="p">=</span> <span class="no">[powershell]</span><span class="p">::</span><span class="n">Create</span><span class="p">().</span><span class="n">AddScript</span><span class="p">(</span><span class="nv">$ScriptBlock</span><span class="p">)</span>
<span class="nv">$Runspace</span><span class="p">.</span><span class="n">RunspacePool</span> <span class="p">=</span> <span class="nv">$RunspacePool</span>
<span class="nv">$Runspaces</span> <span class="p">+=</span> <span class="nb">New-Object</span> <span class="n">PSObject</span> <span class="n">-Property</span> <span class="p">@{</span>
<span class="n">Runspace</span> <span class="p">=</span> <span class="nv">$Runspace</span>
<span class="n">State</span> <span class="p">=</span> <span class="nv">$Runspace</span><span class="p">.</span><span class="n">BeginInvoke</span><span class="p">()</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">while</span> <span class="p">(</span> <span class="nv">$Runspaces</span><span class="p">.</span><span class="n">State</span><span class="p">.</span><span class="n">IsCompleted</span> <span class="o">-contains</span> <span class="nv">$False</span><span class="p">)</span> <span class="p">{</span> <span class="nb">Start-Sleep</span> <span class="n">-Milliseconds</span> <span class="n">10</span> <span class="p">}</span>
<span class="nv">$Results</span> <span class="p">=</span> <span class="p">@()</span>
<span class="nv">$Runspaces</span> <span class="p">|</span> <span class="k">ForEach</span><span class="n">-Object</span> <span class="p">{</span>
<span class="nv">$Results</span> <span class="p">+=</span> <span class="nv">$_</span><span class="p">.</span><span class="n">Runspace</span><span class="p">.</span><span class="n">EndInvoke</span><span class="p">(</span><span class="nv">$_</span><span class="p">.</span><span class="n">State</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div>
<p>This code is executing a code block ten times (just returning a random number), and allowing 5 executions to run at a time via a runspace pool. This code will run just fine, and in many cases you can probably just copy and paste it into your script and be good to go. But that would be a pretty lame way to end a blog post, so lets dig a little deeper and see what all of this code does.</p>
<h2>But First, Pedantics</h2>
<p>So I have a problem with some of the posts I've read about runspaces. It all comes down to a small detail that I think makes a big difference in your understanding of them. </p>
<div class="codehilite"><pre><span></span><code><span class="nv">$Runspace</span> <span class="p">=</span> <span class="no">[powershell]</span><span class="p">::</span><span class="n">Create</span><span class="p">()</span>
</code></pre></div>
<p>This code looks innocent. What does it do? You'd probably think it's creating a new runspace, but it's not. This code is instead creating a fresh instance of PowerShell. If you run this code and run <code>Get-Runspace</code> you'll see there is still just one listed, the one attached to your current session. So what is this <em>instance</em> we just created? </p>
<p>A PowerShell instance handles almost everything about executing PowerShell code, except executing the actual commands. A PowerShell instance is a "wrapper" of sorts that abstracts a lot of the functionality related to the <strong>runspace</strong> that is doing all the work. The PowerShell instance handles creating the command pipeline (think of it like a queue of commands to run) that the runspace will use, and also handles adding commands to it. A quick example script can show how you might do this manually without directly using the instance:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$PowerShell</span> <span class="p">=</span> <span class="no">[powershell]</span><span class="p">::</span><span class="n">Create</span><span class="p">()</span>
<span class="nv">$Pipeline</span> <span class="p">=</span> <span class="nv">$PowerShell</span><span class="p">.</span><span class="n">Runspace</span><span class="p">.</span><span class="n">CreatePipeline</span><span class="p">()</span>
<span class="nv">$Pipeline</span><span class="p">.</span><span class="n">Commands</span><span class="p">.</span><span class="n">Add</span><span class="p">({</span><span class="nb">Get-Variable</span><span class="p">})</span>
<span class="nv">$Pipeline</span><span class="p">.</span><span class="n">Invoke</span><span class="p">()</span>
</code></pre></div>
<p>When you create a new PowerShell instance it comes with a default runspace, in this script we are using that runspace directly to do some work. This approach is pretty verbose though and can get really complicated, so instead we typically use the PowerShell instance itself to do this work:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$PowerShell</span> <span class="p">=</span> <span class="no">[powershell]</span><span class="p">::</span><span class="n">Create</span><span class="p">().</span><span class="n">AddScript</span><span class="p">({</span><span class="nb">Get-Variable</span><span class="p">})</span>
<span class="nv">$PowerShell</span><span class="p">.</span><span class="n">Invoke</span><span class="p">()</span>
</code></pre></div>
<p>The distinction between instances and runspaces isn't important for simple examples, but as we get deeper in future posts it will make it easier to understand more complex examples. Now that we have that out of the way we can dive into the example in a little more depth.</p>
<h2>example explained</h2>
<p>Starting with the first two lines in our example we are creating a runspace pool, and then opening it.</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$RunspacePool</span> <span class="p">=</span> <span class="no">[RunspaceFactory]</span><span class="p">::</span><span class="n">CreateRunspacePool</span><span class="p">(</span><span class="n">1</span><span class="p">,</span> <span class="n">5</span><span class="p">)</span>
<span class="nv">$RunspacePool</span><span class="p">.</span><span class="n">Open</span><span class="p">()</span>
</code></pre></div>
<p>A <strong>runspace pool</strong> is a mechanism to control the number of active runspaces executing at any given time. Think of it as a simple concurrency limiter. A runspace pool is attached to any number of PowerShell instances, and in turn those instances communicate with the pool to ensure only a certain number of runspaces execute code at a time. In this case we are creating a pool with a minimum of 1 executing runspace, and a maximum of 5. If we attempt to execute more they will wait for slots to become available as other runspaces complete their work. </p>
<p>Next we are creating an array to hold our instances as they are created, and then entering into a loop using the PowerShell range operator <code>(1..10)</code>. The range operator is a quick way to generate an iterable array of a given size in PowerShell. In this case this operator is just generating an array with 10 elements in it, integers from 1 to 10, which means the code within the loop will be executed 10 times:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$Instances</span> <span class="p">=</span> <span class="p">@()</span>
<span class="p">(</span><span class="n">1</span><span class="p">..</span><span class="n">10</span><span class="p">)</span> <span class="p">|</span> <span class="k">ForEach</span><span class="n">-Object</span> <span class="p">{</span>
<span class="nv">$Instance</span> <span class="p">=</span> <span class="no">[powershell]</span><span class="p">::</span><span class="n">Create</span><span class="p">().</span><span class="n">AddScript</span><span class="p">({</span><span class="nb">Get-Random</span><span class="p">})</span>
<span class="nv">$Instance</span><span class="p">.</span><span class="n">RunspacePool</span> <span class="p">=</span> <span class="nv">$RunspacePool</span>
<span class="nv">$Instances</span> <span class="p">+=</span> <span class="nb">New-Object</span> <span class="n">PSObject</span> <span class="n">-Property</span> <span class="p">@{</span>
<span class="n">Instance</span> <span class="p">=</span> <span class="nv">$Instance</span>
<span class="n">State</span> <span class="p">=</span> <span class="nv">$Instance</span><span class="p">.</span><span class="n">BeginInvoke</span><span class="p">()</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>Within the loop we:</p>
<ul>
<li>Create a new instance, and add a scriptblock to it: <code>{Get-Random}</code></li>
<li>Bind the instance to the runspace pool we created</li>
<li>Add the new instance to our <code>$Instances</code> array (it's more complicated than this, but we'll discuss it more below)</li>
</ul>
<p>The code we are using to add the instance to the array is a little odd:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$Instances</span> <span class="p">+=</span> <span class="nb">New-Object</span> <span class="n">PSObject</span> <span class="n">-Property</span> <span class="p">@{</span>
<span class="n">Instance</span> <span class="p">=</span> <span class="nv">$Instance</span>
<span class="n">State</span> <span class="p">=</span> <span class="nv">$Instance</span><span class="p">.</span><span class="n">BeginInvoke</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div>
<p>Here we are creating a custom object, <code>PSCustomObject</code>, with two properties:</p>
<ul>
<li><code>instance</code> - This is the new PowerShell instance we created</li>
<li><code>State</code> - This is the output of <code>BeginInvoke()</code></li>
</ul>
<p>When we call <code>BeginInvoke()</code> we are telling the instance to execute its scriptblock <strong>asynchronously</strong>, and return an object we can use to determine the state of that execution. The object returned is an <code>AsyncResult</code> object, this object has an <code>IsCompleted</code> property to tell us if the script is complete or not, and also stores the final results of the execution when it completes.</p>
<p>Now back to the rest of the script. After all of our instances have been added to the array we enter a while loop and use the object we got back from <code>BeginInvoke()</code> to wait until all of our instances have finished executing:</p>
<div class="codehilite"><pre><span></span><code><span class="k">while</span> <span class="p">(</span> <span class="nv">$Instances</span><span class="p">.</span><span class="n">State</span><span class="p">.</span><span class="n">IsCompleted</span> <span class="o">-contains</span> <span class="nv">$False</span><span class="p">)</span> <span class="p">{</span> <span class="nb">Start-Sleep</span> <span class="n">-Milliseconds</span> <span class="n">10</span> <span class="p">}</span>
</code></pre></div>
<p>Specifically we are generating an array of <code>IsCompleted</code> properties for all of the instances we created, and then seeing if that array contains <code>$False</code>, which would indicate something is still running.</p>
<blockquote>
<p>Many examples omit the sleep statement in the while loop. This can lead to lots of extra CPU usage. When you omit the sleep you enter a tight loop where the computer will check the completion state of your instances as fast as it can. Adding the sleep slows this process down and can reduce CPU consumption considerably. In simple tests I have seen scripts go from consuming 25% CPU during this loop to not really consuming any noticable amount at all, just by adding this sleep statement.</p>
</blockquote>
<p>Once everything has completed we break out of our while loop and finally loop through the instances and get our results:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$Results</span> <span class="p">=</span> <span class="p">@()</span>
<span class="nv">$Instances</span> <span class="p">|</span> <span class="k">ForEach</span><span class="n">-Object</span> <span class="p">{</span>
<span class="nv">$Results</span> <span class="p">+=</span> <span class="nv">$_</span><span class="p">.</span><span class="n">Instance</span><span class="p">.</span><span class="n">EndInvoke</span><span class="p">(</span><span class="nv">$_</span><span class="p">.</span><span class="n">State</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div>
<p>To get the results from a completed instance you have to execute the <code>EndInvoke()</code> method. <code>EndInvoke()</code> is kind of a misleading name, it isn't ending anything, instead it is retrieving whatever output was generated by an asynchronous process in a instance. If you recall, when we started the executions on our instances we called <code>BeginInvoke()</code> which returned an <code>AsyncResult</code> object which we then stored in the <code>State</code> property of our <code>$Instances</code> array. So the above code is looping through each of our instances, and calling <code>EndInvoke()</code> for the <code>State</code> property of that instance. It is then taking whatever data is returned and putting into a <code>$Results</code> array for use later.</p>
<blockquote>
<p>While not required by any means, if you want to read a bit more on the async objects being passed around for this to work, take a look at this: <a href="https://docs.microsoft.com/en-us/dotnet/api/system.iasyncresult?redirectedfrom=MSDN&view=netframework-4.8">Microsoft Docs: IAsyncResult</a></p>
</blockquote>
<h2>summary</h2>
<p>In this post we covered a few key concepts related to runspaces and instances:</p>
<ul>
<li><strong>Instance</strong> - A fresh PowerShell child process spawned under the current process</li>
<li><strong>Runspace</strong> - A thread that executes PowerShell code within a PowerShell instance</li>
<li><strong>RunspacePool</strong> - Controls the number of runspaces that can be executing at any given time</li>
<li><strong>BeingInvoke()</strong> - Method that can be called on any instance object to start execution, this method call will return an <code>AsyncResult</code> object that can be used to track completion and obtain the output of the script</li>
<li><strong>IsCompleted</strong> - Property of the <code>AsyncResult</code> object output by <code>BeginInvoke()</code>, this will tell you when an instance has completed execution</li>
<li><strong>EndInvoke()</strong> - Method that can be called on any instance object to end execution and return results - it expects an <code>AsyncResult</code> object as an argument</li>
</ul>
<h2>conclusion</h2>
<p>This was a relatively quick introduce runspaces and instances. Hopefully you've come away with a better understanding of what they do and why you should think about using them. In future posts we'll go into more advanced topics like passing data into your instances, sharing data between instances, and debugging methods. When this series wraps up we will go over a more complex structure I developed to break out of work being done on parallel instances early to allow you to fail fast and not waste time waiting for everything to finish.</p>data céilí and my first pre-con2020-02-14data_ceili_and_first_pre-conA few exciting opportunities for me in the coming months.<p>While I am admittedly a home body, I do get out every once in a while and present at conferences (mostly SQL Saturday events), but this year I have two exciting opportunities that will allow me to branch out a bit. sql server</p>
<h1>Data Céilí</h1>
<p>I'm honored to have been selected to present about SQL Server indexing on the green track at <a href="https://www.dataceili.io/">Data Céilí</a>. Data Céilí (Gaelic for "social visit/gathering") is an exciting new data conference in Dublin Ireland organized in part by my amazing co-worker Andrew Pruski (<a href="https://dbafromthecold.com/">B</a>|<a href="https://twitter.com/dbafromthecold">T</a>). The green track is interesting in that it is a remote track, the speakers will all be presenting remotely. While I wish I could make it out to Ireland to present in person, I'm just thankful to be a part of this event. I love seeing conferences trying new things and I think the green track is a great idea. I'm excited to see how this track goes and how this conference grows in the future. Read more about the conference <a href="https://www.dataceili.io/">here</a> (it's not too late to sign up!).</p>
<h1>SQL Saturday Raleigh - PowerShell Pre-Con</h1>
<p>SQL Saturday Raleigh is coming up in April and I'm proud to say that my pre-con submission "<a href="https://www.eventbrite.com/e/powershell-top-to-bottom-by-mark-wilkinson-sql-saturday-raleigh-2020-tickets-94553803973">PowerShell Top to Bottom</a>" was accepted. This will not be my first time presenting about PowerShell, but it will be my first pre-con. It's exciting and scary at the same time, but I am really looking forward to having and entire day to focus on such a great topic. This pre-con is designed to give folks the knowledge they need to start building custom functions and modules with PowerShell. I'm covering everything from the basics to using runspaces. I know a ton of people are using PowerShell in their day-to-day work, but I think a smaller percentage would be able to fix issues if they arose, or tweak a script they downloaded to better fit their needs. My hope is that this pre-con will get people to the point where they can start doing that and potentially even start contributing their work to the community at large. You can read more about the pre-con, and the SQL Saturday event, at the <a href="https://www.sqlsaturday.com/981/eventhome.aspx">SQL Saturday Raleigh site</a>. </p>
<p>I look forward to seeing (or at least talking in the case of Data Céilí) with folks at both events, sign up while you still can!</p>handling unicode in powershell2020-02-19unicode_powershellHow to properly handle unicode characters in PowerShell.<p>This post is inspired by an odd situation I ran into in a project I'm working on. I have the need to pull specific revisions of files out of a git repository, save those files, and then execute the contents. This all worked fine until it didn't. I received some complaints that unicode characters in the files we getting mangled, and sure enough they were. But why? In this post I'll explain what happened to me, and ways you can avoid it yourself.</p>
<p>In the examples below we are going to be working with a file called "PoShUnicodeSample.txt" that contains the following:</p>
<div class="codehilite"><pre><span></span><code>Here is some text with a Unicode character embedded: ⁋
</code></pre></div>
<blockquote>
<p><strong>NOTE</strong> The issue we are discussing in this post seems to be specific to Windows, Linux does not have the same behavior, but everything we are talking about will work on any OS.</p>
</blockquote>
<h1>Command Specific Encodings</h1>
<p>Many commands in PowerShell will take <code>-Encoding</code> as a parameter. For example, if you want to read a file into a variable, and that file has unicode characters, the following will result in mangled data:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$data</span> <span class="p">=</span> <span class="nb">Get-Content</span> <span class="n">-Path</span> <span class="s2">"PoShUnicodeSample.txt"</span>
<span class="nv">$data</span> <span class="p">|</span> <span class="nb">Out-File</span> <span class="n">-FilePath</span> <span class="s2">"Temp.txt"</span>
</code></pre></div>
<p>If we open "Temp.txt" we'll see the following:</p>
<div class="codehilite"><pre><span></span><code>Here is some text with a Unicode character embedded: â‹
</code></pre></div>
<p>Luckily we can fix this with <code>Encoding</code>!</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$data</span> <span class="p">=</span> <span class="nb">Get-Content</span> <span class="n">-Path</span> <span class="s2">"PoShUnicodeSample.txt"</span> <span class="n">-Encoding</span> <span class="n">UTF8</span>
<span class="nv">$data</span> <span class="p">|</span> <span class="nb">Out-File</span> <span class="n">-FilePath</span> <span class="s2">"Temp.txt"</span>
</code></pre></div>
<p>Tada! We now have a proper unicode encoded output file, right? Almost. If you open the file in a text editor like VSCode it reports the file as being encoded in <code>UTF16LE</code>. If you look at the <code>Out-File</code> documentation you'll see the default output encoding is <code>UTF8NoBOM</code>. If we want straight UTF-8 we have to tell it to use that encoding via <code>-Encoding UTF8</code>.</p>
<p>So, if you are working with unicode, and the encoding is important, make sure you are always setting the encoding explicitly. When I was troubleshooting this issue, I thought this solved my issue, but when I put the changes into the project I was working on, I was still seeing the issue. It took a little help from the folks in #PowershellHelp on the <a href="https://sqlcommunity.slack.com">SQL Community Slack</a> to get the issue solved.</p>
<h1>Default Encodings</h1>
<p>PowerShell has a set of default encodings it uses for all input and output operations. You can check what your current settings are by looking at the <code>OutputEncoding</code> and <code>InputEncoding</code> property of the console:</p>
<div class="codehilite"><pre><span></span><code>PS> [Console]::OutputEncoding
IsSingleByte : True
BodyName : iso-8859-1
EncodingName : Western European (Windows)
HeaderName : Windows-1252
WebName : Windows-1252
WindowsCodePage : 1252
IsBrowserDisplay : True
IsBrowserSave : True
IsMailNewsDisplay : True
IsMailNewsSave : True
EncoderFallback : System.Text.InternalEncoderBestFitFallback
DecoderFallback : System.Text.InternalDecoderBestFitFallback
IsReadOnly : True
CodePage : 1252
PS> [Console]::InputEncoding
IsSingleByte : True
BodyName : iso-8859-1
EncodingName : Western European (Windows)
HeaderName : Windows-1252
WebName : Windows-1252
WindowsCodePage : 1252
IsBrowserDisplay : True
IsBrowserSave : True
IsMailNewsDisplay : True
IsMailNewsSave : True
EncoderFallback : System.Text.InternalEncoderBestFitFallback
DecoderFallback : System.Text.InternalDecoderBestFitFallback
IsReadOnly : True
CodePage : 1252
</code></pre></div>
<p>As you can see, on my system, the default encoding is <code>iso-8859-1</code>. Yours may be different, and if you are using a Linux system it most certainly will be (it will likely be UTF-8 in that case). </p>
<h1>Solving my Problem</h1>
<p>As I said at the top of this post, when I encountered this issue I was using <code>git show</code> to pull the content of a script file from a git repo and store it in a local file. the following syntax will accomplish that:</p>
<div class="codehilite"><pre><span></span><code><span class="n">git</span> <span class="n">show</span> <span class="s2">"origin/Branch:path/to/file.txt"</span> <span class="p">|</span> <span class="nb">Out-File</span> <span class="n">-FilePath</span> <span class="s2">"LocalFile.txt"</span> <span class="n">-Encoding</span> <span class="s2">"utf8"</span>
</code></pre></div>
<p>But I found that the unicode characters were STILL being mangled. This is because the default output of the console was not <code>UTF-8</code>, so any commands executed in that console would output to the <code>iso-8859-1</code> encoding. This includes non-powershell commands, like <code>git</code>. To fix this, we have to change the default encoding of the console to UTF-8:</p>
<div class="codehilite"><pre><span></span><code><span class="n">PS</span><span class="p">></span> <span class="no">[Console]</span><span class="p">::</span><span class="n">OutputEncoding</span> <span class="p">=</span> <span class="no">[System.Text.Encoding]</span><span class="p">::</span><span class="n">UTF8</span>
</code></pre></div>
<p>Re-running my <code>git show</code> command after that results in the unicode characters being preserved. Success! </p>
<p>If you are always executing scripts under your own PowerShell console, and want to make sure you are always handling unicode data properly, you could add the following to your PowerShell profile:</p>
<div class="codehilite"><pre><span></span><code><span class="no">[Console]</span><span class="p">::</span><span class="n">OutputEncoding</span> <span class="p">=</span> <span class="no">[System.Text.Encoding]</span><span class="p">::</span><span class="n">UTF8</span>
<span class="no">[Console]</span><span class="p">::</span><span class="n">InputEncoding</span> <span class="p">=</span> <span class="no">[System.Text.Encoding]</span><span class="p">::</span><span class="n">UTF8</span>
</code></pre></div>
<p>That combined with the <code>-Encoding</code> parameter used when working with files should cover most of your needs. If you are working in an environment where you don't have access to the profile you'll just have to make sure to include the console encoding changes in your scripts.</p>
<h1>Conclusion</h1>
<p>Overall PowerShell offers a lot of flexibility around handling different file encodings. Unfortunately it's not overly obvious what encoding you'll end up with if you don't set them explicitly.</p>sql server query store - two stories2020-03-11qds_productionTrials and tribulations of running QDS in production.<p>This post is a part of the SQL Server community's T-SQL Tuesday event. This month is being hosted by <a href="https://tracyboggiano.com/archive/2020/03/t-sql-tuesday-124-using-query-store-or-not-lets-blog/">Tracy Boggiano</a>. Thanks Tracy!</p>
<p>When the Query Data Store (QDS) feature was announced for SQL Server 2016, we were excited about the prospect of being able to have deep insight on any query running in our environment. I work for a company that deals heavily in the e-commerce space, and we have a large SQL Server footprint. Our environment is unique in that is essentially a multi-tenant system, but all the tenants could have wildly different workloads. It's really the kind of query execution scenario QDS was built for. We had the pleasure of working with the Microsoft SQLCAT team to get 2016 and QDS up and running in our production environment before it was GA. </p>
<p>In this post I'm going to share two stories about our QDS experience (from pre and post GA of the feature). One from the perspective of the Database Developer, and one from the Database Administrator. For the most part this is not a technical post full of queries and code samples. It's just me talking about some things I have experienced using QDS in production.</p>
<blockquote>
<p>DISCLAIMER - These are MY observations. Your story may be different, and in many respects I hope it was. We worked closely with Microsoft to roll out 2016 before it was GA, and after to make sure things were running smoothly. It just turns out that our systems stretch the limitations of a lot of products, and SQL Server is no exception.</p>
</blockquote>
<h1>The Developer</h1>
<p>Our initial interactions with QDS were nothing short of amazing. We now had full details about every query that was being executed against our instances. This came with some immediate benefits that created actionable tasks. We could start asking questions like "do we need to run this query 2 million times per minute?" and "when did this procedure start performing poorly?". </p>
<p>Since those early days, QDS has gotten more and more useful for us. The addition of Automatic Plan Regression Correction (APRC) was huge, showing us upwards of 15% CPU usage reduction in a very short period of time. Later they added wait information as well, which allowed us to see what waits specific plans typically accumulated. Overall it has given us all the information we could ever need to to troubleshoot performance issues at the database level, but there lyes the rub. QDS is database specific, which is not super helpful when you are looking at the performance of thousands of databases.</p>
<p>Without a consolidated view of our total environment, or even a whole server, QDS data was starting to become less useful. Querying the data by hand also presented some challenges, as it seems the data was hyper-optimized for INSERTs, but not for SELECTs. Because of this, running huge SELECT statements across all databases on a server can pretty expensive, especially if you are doing it ad-hoc. Enter CenteralQDS. </p>
<h2>CentralQDS</h2>
<p>With no obvious communications from Microsoft about a future "full server QDS", a member of my team embarked on a project to create an environment-wide QDS system. With the help of our development team (a true DBA/Dev collaboration project that any manager would be proud of) we now have a fully centralized view of all QDS data gathered across our hundreds of servers and thousands of databases. With the help of a PowerBI front-end, anyone can now ask even more important questions than the original "when did this procedure start performing poorly?"; instead they can ask questions like "do we have any procedures that aren't executed anymore?" or "what is the most expensive procedure in production?". This system has revolutionized how we troubleshoot performance. In most cases it has even moved performance troubleshooting away from my team (the DBAs) and back to the developers themselves.</p>
<h2>Conclusions for the Developer</h2>
<p>From a developer standpoint QDS is an outstanding feature (even if it has some database-centric short-comings). It has changed how troubleshooting is done and also allows for more "big picture" planning and analysis. Without having a full-environment view it can be less useful, but the fact that the data is there to aggregate and analyse is very very useful.</p>
<h1>The Administrator</h1>
<p>The developer story of QDS is a good one. The administrator story is less so. If you are looking at an ancient map, and part of the map is labeled "there be dragons here", QDS is beyond that, in the part of the world that folks hadn't dared go before. While QDS has been a boon for performance improvement projects, it has a lot of hidden impact on your instances. I struggled with how best to write this section and decided to first present the catastrophic failures we had with QDS, followed by a break down by issue.</p>
<h2>The Perfect Storm</h2>
<p>Going to production with a CTP version of SQL Server can be a bit nerve wracking to be honest. In testing SQL Server 2016 was solid and we didn't see any issues except maybe the odd query regression. As anyone supporting production systems knows though, all the testing and planning in the world tend to crumble under the weight of production.</p>
<p>A month or so after upgrading production to SQL Server 2016 we started to see breathtaking (it's the only word clean enough to use here) amounts of tempdb contention on a number of our higher-volume instances. We struggled to find a root cause of the contention, but assumed there had been a workload change due to a recent release (there was a release the day before this started happening). We couldn't pin down the exact change, but figured it had to be related. When the contention would occur, the entire instance would become unresponsive, to the point where you couldn't even connect to it. Luckily an AG failover to the secondary would still work and seemed to clear the contention. </p>
<p>After further investigation we discovered that contention was happening on the base system tables in tempdb; it wasn't standard PFS contention. This made even less sense. Based on the tables involved it pointed at auto-stats potentially being the issue. Whenever we saw the issue we usually saw contention on the base table that stores stats objects for tempdb. This was the beginning of a chain of trial and error that lead us to suspect a lot of different issues:</p>
<ul>
<li>Too much tempdb usage</li>
<li>Large sorting operations spilling into tempdb</li>
<li>Index maintenance operations using tempdb</li>
<li>Large table variables and other objects that can use tempdb on the backend</li>
</ul>
<p>In the end though, we finally discovered that it was none of that, and all of it. It turns out that in 2016 a change was made that resulted in ~100% MORE <code>PAGELATCH</code> waits when creating objects in tempdb. This combined with a few other issues to result in what I came to call "pagelatch storms". Eventually Microsoft released a fix for the <code>PAGELATCH</code> issue which brought things back to pre-2016 levels (<a href="https://support.microsoft.com/en-us/help/4013999">KB4013999</a>). So what were these other issues? Here's a list:</p>
<ul>
<li><strong>Size-based cleanup</strong> - The size-based cleanup process in QDS is BAD. On systems with a lot of unique queries it can consume upwards of 70% CPU (this is on boxes with 8+ cores) on an instance, especially if multiple cleanups are running. This is mostly because QDS is designed to handle lots of writes, but reads are expensive. Whenever size-based cleanup runs on a database with lots of unique queries, it will likely spill into tempdb. Combine that spilling with the increase in <code>PAGELATCH</code> waits in 2016 and you already have an issue. Combine it with the others issues below and it can bring down a server.<blockquote>
<p>Look for future posts about some custom QDS clean-up scripts we run to avoid the size-based clean-up operations.</p>
</blockquote>
</li>
<li><strong>Lots of tempdb usage</strong> - We use a lot of tempdb, and when the issue would occur, anything that used tempdb was dead in it's tracks.</li>
<li><strong>Auto-stats</strong> - While the issue was occurring auto stats updates would get blocked and then end up blocking other operations, adding to the chaos.</li>
<li><strong>Missing and misplaced information</strong> - When QDS was first released the QDS operations were hidden from <code>dm_exec_requests</code>, so we had no way of telling what was happening. Beyond that, a lot of the resource consumption for QDS occurs in the <code>Default</code> resource pool, which again masked the issues.</li>
<li><strong>Timing</strong> - These issues didn't start until QDS started hitting max size, so that was over a month in some cases, and it didn't hit all instances and databases at the same time. This means the issue seemed "random" when it was happening. Not only was it random, in some cases (which we still see today) the clean-up would just happen to fire off for multiple databases at a time, further adding to contention.</li>
</ul>
<h1>Conclusion</h1>
<p>Overall QDS is an amazing feature worth using. Like any new technology though, you have to be on the lookout for unexpected issues. I wish Microsoft would release their own Central QDS system, as I think it would make it infinitely more useful for larger shops. When these issues first started cropping up it made me question installing CTPs in production, but honestly we would have been bitten by this issue regardless. If you are thinking about using QDS, you just need to remember a few things:</p>
<ul>
<li>Start with the default settings and adjust as needed</li>
<li>Keep an eye on how much of the allocated QDS space is being consumed to avoid the size-based clean-up</li>
<li>The more unique your queries, the quicker space will fill up</li>
</ul>eightkb2020-04-12eightkbI'm helping organize a SQL Server internals conference!<p>I'm excited and a little scared to announce that (with the help of some friends) I'm helping launch my first conference! It's a team effort with Andrew Pruski, Anthony Nocentino, and myself. So far it's been a great experience and the community response has been amazing. We've already got a fantastic set of speakers that have submitted talks with more to come!</p>
<h2>EightKB</h2>
<p>EightKB is a virtual mini-conference focusing on SQL Server internals. We'll be hosting 5-7 sessions on internals topics from 45 to 75 minutes long. I've been to a few conferences and SQL Saturdays in my time and have always wished I could attend more sessions on database internals, I'm talking about things like SQLOS memory architecture talks, and how DBCC commands work under the covers. Internals just isn't the focus of most events though (with good reason), so we wanted to launch our own. </p>
<p>Besides wanting to organize a conference around the things I'm interested in I also thought folks would appreciate a highly technical conference during quarantine. It helps me stay busy as well. I have four children so I'm usually pretty busy anyhow, but being able to dedicate some time to an event I think the SQL community could really benefit from has been great. I don't get to give back to the community that often so I'm excited to be a part of this.</p>
<h2>Join Us</h2>
<p>If an internals conference (with session levels from <code>300</code> to <code>Insanity</code>!!) sounds fun to you, join us on June 17th. You can learn more at the EightKB site: <a href="https://eightkb.online">https://EightkB.online</a>. See you there!</p>the chasm2020-04-18the_chasmSelf-doubt, the silent hero of production.<p>My last post was from before the world exploded. The current events related to COVID-19 have made it tough to write technical posts, so I'm going to mix it up a bit and talk about something else: the chasm of self-doubt.</p>
<p>This post was inspired by a conversation I had with my team the other morning. We were troubleshooting a partition-based GC process and after looking at it a while folks started doubting what they knew about partitioning. This is common in our line of work. Things in the SQL Server world change quickly, and it's not uncommon for old myths to be taken as truths (like table variables being stored completely in memory). I think for an outsider, this amount of self-doubt and questioning might look unproductive and unhealthy, but I have a different take:</p>
<blockquote>
<p>Ideally, all engineers of any kind should always have their legs dangling into the chasm, at a minimum. It's the only thing that protects production.</p>
</blockquote>
<h1>falling in</h1>
<p>Falling into the chasm can feel like an uncontrollable downward spiral. You will find yourself researching things you thought you knew, and often finding that you were right. Sometimes though you <em>do</em> discover that you weren't 100% correct about something, but that small misunderstanding hasn't had much impact on your work. Other times you may find that you were entirely wrong about something. This can be a bit crushing, but the important thing is to acknowledge what you find and learn from it. Nobody can get everything right all the time, and as long as you learn along the way, things will be fine.</p>
<h1>dangling your feet</h1>
<p>Back to my quote above, I think self-doubt is extremely healthy in an engineering team. Without a constant low-level feeling of self-doubt you can start to get cocky. Think about that period maybe 2 years into your current profession. You've learned a lot but haven't yet learned enough to understand how dangerous you are. If you have a good amount of self-confidence you'll likely cause a production outage that will naturally instill you with that fear and self-doubt. If not, it might take a bit longer to get there. Obviously everyone is different, and I've certainly met talented engineers that started out and already had this self-doubt. The thing to remember is that it's ok, and can even be helpful to have these feelings.</p>
<h1>to new engineers</h1>
<p>This may be wrong, but I have a hard time trusting folks that don't experience that fear and doubt. Most of the time it is absolutely not the fault of the engineer, it's just a matter of circumstance. If you are new and are experiencing situations where you feel folks don't trust what you are doing, just remember that a lot of those people have probably been burned in the past by an over-abundance of confidence. To a lot of us this self-doubt is the only thing protecting production.</p>
<p>You can have all the controls in the world in place, but things will always break in production. Knowing that, you have to plan as much as possible and approach solutions with caution and care. If you encounter situations where you think folks don't trust your judgement, take a second and try to have a conversation about what could go wrong with the idea you are proposing; ask others for input. Like with most things, communication is key here. If you can show folks that you have some doubts, and can show how you think you've mitigated the risks, you'll build trust a lot faster.</p>
<h1>living with it</h1>
<p>Self-doubt is just a part of life as an engineer. It will always be there. Just know that it's normal, and can play an important role in keeping your production systems running. As engineers we should be talking about this, and encouraging conversations about it. Phrases like "I don't know" and "I need to double check that" should be welcome, and if you have doubts they should be expressed and addressed. Resolving your doubts as a team can be a great way to increase trust on your teams and make people more comfortable sharing their ideas. It can also be a great training tool, as the whole team can learn from eachother. So next time you have some doubts, say it out loud (or via chat) and get a conversation started, you and your team will only be stronger for it.</p>optional parameters with splatting in powershell2020-08-13optional_param_splattingHow to use splatting to better handle optional parameters when calling PowerShell functions<p>Today's post is a quick one. This post was inspired by a conversation I had in the <code>#powershellhelp</code> channel on the <a href="https://sqlcommunity.slack.com">SQL Community Slack</a>. Splatting is an underused feature of PowerShell. It can make your code easier to read, and provide a lot of flexibility when dealing with things like optional parameters. In this post we'll quickly go over <em>what</em> splatting is, and how it can make your code easier to write and easier to read.</p>
<h1>Splatting</h1>
<p>I have to start off by saying I hate the name "splatting". I didn't come up with it, and I don't like using it, but it's the only word we have. Splatting is a way to pass parameter values to a function using a single <code>array</code> or <code>hashtable</code>. In this post we'll be talking about hashtables because I think it is the more useful of the two. </p>
<p>Splatting is easy to explain in an example. Let's say you are executing some TSQL code against a SQL Server. More than likely you'll be using <code>Invoke-SqlCmd</code>. Your typical call will look like this:</p>
<div class="codehilite"><pre><span></span><code><span class="nb">Invoke-SqlCmd</span> <span class="n">-ServerInstance</span> <span class="n">myserver</span><span class="p">.</span><span class="n">markw</span><span class="p">.</span><span class="n">dev</span> <span class="n">-Query</span> <span class="s2">"SELECT @@VERSION"</span> <span class="n">-Database</span> <span class="s2">"master"</span>
</code></pre></div>
<p>That's pretty simple, but lets see how this looks using splatting:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$Params</span> <span class="p">=</span> <span class="p">@{</span>
<span class="n">ServerInstance</span> <span class="p">=</span> <span class="s1">'myserver.markw.dev'</span>
<span class="n">Query</span> <span class="p">=</span> <span class="s2">"SELECT @@VERSION"</span>
<span class="n">Database</span> <span class="p">=</span> <span class="s1">'master'</span>
<span class="p">}</span>
<span class="nb">Invoke-Sqlcmd</span> <span class="nv">@Params</span>
</code></pre></div>
<p>That's it! That's splatting. You can already see how it might make things easier to read, especially if you are calling a function with a lot of parameters. Where splatting really shines though, is optional and conditional parameters.</p>
<h1>Optional and conditional parameters with splatting</h1>
<p>Let's say we want to use the example above, but this time we want to pass in SQL auth credentials if they are given. Without splatting we'd probably end up with something like this:</p>
<div class="codehilite"><pre><span></span><code><span class="k">if</span> <span class="p">(</span> <span class="nv">$SqlUser</span> <span class="o">-and</span> <span class="nv">$SqlPass</span><span class="p">)</span> <span class="p">{</span>
<span class="nb">Invoke-SqlCmd</span> <span class="n">-ServerInstance</span> <span class="n">myserver</span><span class="p">.</span><span class="n">markw</span><span class="p">.</span><span class="n">dev</span> <span class="p">`</span>
<span class="n">-Query</span> <span class="s2">"SELECT @@VERSION"</span> <span class="p">`</span>
<span class="n">-Database</span> <span class="s2">"master"</span> <span class="p">`</span>
<span class="n">-Username</span> <span class="nv">$SqlUser</span> <span class="p">`</span>
<span class="n">-Password</span> <span class="nv">$SqlPass</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nb">Invoke-SqlCmd</span> <span class="n">-ServerInstance</span> <span class="n">myserver</span><span class="p">.</span><span class="n">markw</span><span class="p">.</span><span class="n">dev</span> <span class="p">`</span>
<span class="n">-Query</span> <span class="s2">"SELECT @@VERSION"</span> <span class="p">`</span>
<span class="n">-Database</span> <span class="s2">"master"</span>
<span class="p">}</span>
</code></pre></div>
<p>If you ever want to add a <code>QueryTimeout</code> parameter to this, or change the database, you'll have to alter two function calls. Lets see how this would work with splatting:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$Params</span> <span class="p">=</span> <span class="p">@{</span>
<span class="n">ServerInstance</span> <span class="p">=</span> <span class="s1">'myserver.markw.dev'</span>
<span class="n">Query</span> <span class="p">=</span> <span class="s2">"SELECT @@VERSION"</span>
<span class="n">Database</span> <span class="p">=</span> <span class="s1">'master'</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$SqlUser</span> <span class="o">-and</span> <span class="nv">$SqlPass</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$Params</span><span class="p">.</span><span class="n">Username</span> <span class="p">=</span> <span class="nv">$SqlUser</span>
<span class="nv">$Params</span><span class="p">.</span><span class="n">Password</span> <span class="p">=</span> <span class="nv">$SqlPass</span>
<span class="p">}</span>
<span class="nb">Invoke-SqlCmd</span> <span class="nv">@Params</span>
</code></pre></div>
<p>This is a lot cleaner, and if you ever want to add more parameters you can just add them to the <code>$Params</code> hashtable once and be done. One important thing to note here is that when we use splatting, we don't pass the hashtable in using the typical <code>$</code>, we have to use <code>@</code> so PowerShell knows we are using splatting. </p>
<h1>Partial parameters</h1>
<p>One more thing you can do with splatting is only pass in a few parameters, while still manually setting the others. Lets use our above example again:</p>
<div class="codehilite"><pre><span></span><code><span class="c"># Create an empty hashtable</span>
<span class="nv">$Params</span> <span class="p">=</span> <span class="p">@{}</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$SqlUser</span> <span class="o">-and</span> <span class="nv">$SqlPass</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$Params</span><span class="p">.</span><span class="n">Username</span> <span class="p">=</span> <span class="nv">$SqlUser</span>
<span class="nv">$Params</span><span class="p">.</span><span class="n">Password</span> <span class="p">=</span> <span class="nv">$SqlPass</span>
<span class="p">}</span>
<span class="nb">Invoke-SqlCmd</span> <span class="nv">@Params</span>
<span class="n">-ServerInstance</span> <span class="s1">'myserver.markw.dev'</span> <span class="p">`</span>
<span class="n">-Query</span> <span class="s2">"SELECT @@VERSION"</span> <span class="p">`</span>
<span class="n">-Database</span> <span class="s1">'master'</span>
</code></pre></div>
<p>In this example we are just using splatting to pass in credentials. If the <code>$SqlUser</code> or <code>$SqlPass</code> variables were not set, the hashtable would be left empty and this is absolutely fine. If the hashtable is empty the function will simply ignore it.</p>
<h1>Conclusion</h1>
<p>I hope you enjoyed the post. Splatting is a powerful feature and this post just scratches the surface of what you can do. If you want to read a bit more, check out the docs over at Microsoft: <a href="https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_splatting?view=powershell-7">About Splatting</a></p>user input and menus in powershell2020-08-17powershell_user_input_menuHow to get input from the user in your PowerShell scripts and create interactive menus.<p>Fully automated hands-off PowerShell scripts can be extermely useful for the DBA or System Administrator, but what if you need to get input from the user, or maybe you want to implement a menu system? Like most things related to PowerShell, you have a few options:</p>
<ul>
<li><code>Read-Host</code></li>
<li><code>[Console]</code> object methods</li>
</ul>
<p>Most use cases are covered by <code>Read-Host</code>, but if you need something a little more flexible, the <code>[Console]</code> methods might be the way to go.</p>
<h1>Read-Host</h1>
<p><code>Read-Host</code> is a built-in cmdlet that blocks the current process waiting for user input followed by a press of the enter key. This can useful if you need to get something like a file path from a user. Here is a pattern I use often:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$MaxRetry</span> <span class="p">=</span> <span class="n">5</span>
<span class="nv">$CurrentAttempt</span> <span class="p">=</span> <span class="n">0</span>
<span class="c"># Get a valid path from a user</span>
<span class="k">While</span> <span class="p">(</span> <span class="nv">$CurrentAttempt</span> <span class="o">-lt</span> <span class="nv">$MaxRetry</span> <span class="o">-and</span> <span class="o">-not</span> <span class="nv">$UserInput</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$UserInput</span> <span class="p">=</span> <span class="nb">Read-Host</span> <span class="n">-Prompt</span> <span class="s2">"Please enter a valid file path"</span>
<span class="k">if</span> <span class="p">(</span> <span class="o">-not</span> <span class="p">(</span><span class="nb">Test-Path</span> <span class="n">-Path</span> <span class="nv">$UserInput</span><span class="p">)</span> <span class="p">)</span> <span class="p">{</span>
<span class="nb">Write-Host</span> <span class="s2">"File path </span><span class="p">$(</span><span class="nv">$UserInput</span><span class="p">)</span><span class="s2"> not found, please try again."</span>
<span class="nv">$CurrentAttempt</span> <span class="p">+=</span> <span class="n">1</span>
<span class="nv">$UserInput</span> <span class="p">=</span> <span class="nv">$False</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span> <span class="o">-not</span> <span class="nv">$UserInput</span> <span class="p">)</span> <span class="p">{</span>
<span class="nb">Write-Error</span> <span class="n">-Message</span> <span class="s2">"Valid file path not entered."</span> <span class="n">-ErrorAction</span> <span class="n">Stop</span>
<span class="p">}</span>
</code></pre></div>
<p><code>Read-Host</code> is simple to use, it only has two parameters: <code>-Prompt</code> and <code>-AsSecureString</code>. The first is used to set the user prompt, the second to capture a password or other sensitive data (outputing it as a SecureString object). If you are a formatting nut like me, it's important to know that <code>Read-Host</code> slaps a <code>:</code> on the end of the prompt. This always bothers me, because I like total control over output whenever possible. For example:</p>
<div class="codehilite"><pre><span></span><code><span class="n">PS</span><span class="p">></span> <span class="nb">Read-Host</span> <span class="n">-Prompt</span> <span class="s2">"How are you feeling today?"</span>
<span class="n">How</span> <span class="n">are</span> <span class="n">you</span> <span class="n">feeling</span> <span class="n">today</span><span class="p">?:</span>
</code></pre></div>
<p>So as you can see, <code>Read-Host</code> is very straight forward. It just reads user input and then outputs it when the user presses enter. </p>
<h1>[Console]</h1>
<p><code>[Console]</code> has a few methods available around keyboard input that make it a lot more flexible and give you a lot more control. The methods we are going to focus on are:</p>
<ul>
<li><code>ReadLine</code></li>
<li><code>ReadKey</code></li>
<li><code>KeyAvailable</code> - This is not a method, but a property of <code>[Console]</code></li>
</ul>
<h2>ReadLine</h2>
<p>First let's look at <code>ReadLine</code>. The docs are pretty great on this method, giving quite a few good examples of how it's used: <a href="https://docs.microsoft.com/en-us/dotnet/api/system.console.readline?view=netcore-3.1">Console.ReadLine Method</a>. Note that the docs are not PowerShell specific, as <code>[Console]</code> is a .Net object, and not something specifically built for PowerShell.</p>
<p>Let's see how our above example would look using <code>[Console]::ReadLine()</code></p>
<div class="codehilite"><pre><span></span><code><span class="nv">$MaxRetry</span> <span class="p">=</span> <span class="n">5</span>
<span class="nv">$CurrentAttempt</span> <span class="p">=</span> <span class="n">0</span>
<span class="c"># Get a valid path from a user</span>
<span class="k">While</span> <span class="p">(</span> <span class="nv">$CurrentAttempt</span> <span class="o">-lt</span> <span class="nv">$MaxRetry</span> <span class="o">-and</span> <span class="o">-not</span> <span class="nv">$UserInput</span><span class="p">)</span> <span class="p">{</span>
<span class="nb">Write-Host</span> <span class="s2">"Please enter a valid file path: "</span> <span class="n">-NoNewline</span>
<span class="nv">$UserInput</span> <span class="p">=</span> <span class="no">[Console]</span><span class="p">::</span><span class="n">Readline</span><span class="p">()</span>
<span class="k">if</span> <span class="p">(</span> <span class="o">-not</span> <span class="p">(</span><span class="nb">Test-Path</span> <span class="n">-Path</span> <span class="nv">$UserInput</span><span class="p">)</span> <span class="p">)</span> <span class="p">{</span>
<span class="nb">Write-Host</span> <span class="s2">"File path </span><span class="p">$(</span><span class="nv">$UserInput</span><span class="p">)</span><span class="s2"> not found, please try again."</span>
<span class="nv">$CurrentAttempt</span> <span class="p">+=</span> <span class="n">1</span>
<span class="nv">$UserInput</span> <span class="p">=</span> <span class="nv">$False</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span> <span class="o">-not</span> <span class="nv">$UserInput</span> <span class="p">)</span> <span class="p">{</span>
<span class="nb">Write-Error</span> <span class="n">-Message</span> <span class="s2">"Valid file path not entered."</span> <span class="n">-ErrorAction</span> <span class="n">Stop</span>
<span class="p">}</span>
</code></pre></div>
<p>So this looks about the same as before with two small differences. First, we are using <code>Write-Host</code> to manually display the prompt (and using <code>-NoNewline</code> to make it feel more like a prompt). This gives us a lot more control over the formatting and even the color of the prompt we display, which is a win in my book. Second, we have simply replaced <code>Read-Host</code> with <code>[Console]::Readline()</code> which will accept user input until the user presses enter.</p>
<h2>ReadKey</h2>
<p>Now let's check out <code>[Console]::ReadKey()</code>. In this case we are going to ask the user to "Press any key to continue..." to keep it simple:</p>
<div class="codehilite"><pre><span></span><code><span class="nb">Write-Host</span> <span class="s2">"Press any key to continue..."</span>
<span class="nv">$null</span> <span class="p">=</span> <span class="no">[Console]</span><span class="p">::</span><span class="n">ReadKey</span><span class="p">(</span><span class="s1">'NoEcho'</span><span class="p">)</span>
</code></pre></div>
<p>By default, <code>ReadKey</code> will echo the users keystroke back to the console, we can supress this via the <code>'NoEcho'</code> option. It will also return information about the keypress which we are just tossing into $null since we don't need them. This example isn't very exciting, but you can already see that using this method gives you control you simply don't have using <code>Read-Host</code>. </p>
<p>To make things a bit more useful, let's take a closer look at the output of <code>ReadKey()</code>. In this quick snippet I am just going to press Shift + M:</p>
<div class="codehilite"><pre><span></span><code><span class="n">PS</span><span class="p">></span> <span class="no">[Console]</span><span class="p">::</span><span class="n">ReadKey</span><span class="p">()</span>
<span class="n">M</span>
<span class="n">KeyChar</span> <span class="n">Key</span> <span class="n">Modifiers</span>
<span class="p">-------</span> <span class="p">---</span> <span class="p">---------</span>
<span class="n">M</span> <span class="n">M</span> <span class="n">Shift</span>
</code></pre></div>
<p>First, it shows the character I typed (an uppercase M), if I would have added <code>'NoEcho'</code> this would have been supressed. Then it outputs an object that contains:</p>
<ul>
<li>the character I typed <code>KeyChar</code></li>
<li>the key I pressed <code>Key</code></li>
<li>the modifier key(s) I used <code>Modifiers</code></li>
</ul>
<p>This is pretty cool as it would allow us to create menu systems with case sensitivity, or even implement special shortcut combos in a more interactive script. </p>
<h1>A Simple Menu</h1>
<p>Let's look at a more complicated example. Let's say you want to display a menu of choices and you want the user to choose one using a single letter:</p>
<div class="codehilite"><pre><span></span><code><span class="c"># First define some menu options you want to display</span>
<span class="c"># The underscore highlights the character the user would type</span>
<span class="nv">$Choices</span> <span class="p">=</span> <span class="p">@(</span>
<span class="s2">"_Continue"</span>
<span class="s2">"_Abort"</span>
<span class="s2">"_Burn it all down"</span>
<span class="p">)</span>
<span class="nv">$MenuChoices</span> <span class="p">=</span> <span class="p">@()</span>
<span class="k">Foreach</span> <span class="p">(</span> <span class="nv">$Choice</span> <span class="k">in</span> <span class="nv">$Choices</span> <span class="p">)</span> <span class="p">{</span>
<span class="c"># This finds the underscore and finds the following character</span>
<span class="c"># so we can use it in our menu</span>
<span class="nv">$ChoiceCharacter</span> <span class="p">=</span> <span class="nv">$Choice</span> <span class="o">-replace</span> <span class="s1">'.*_(.).*'</span><span class="p">,</span><span class="s1">'$1'</span>
<span class="nv">$MenuChoices</span> <span class="p">+=</span> <span class="p">@{</span>
<span class="s1">'Key'</span> <span class="p">=</span> <span class="nv">$ChoiceCharacter</span>
<span class="s1">'MenuItem'</span> <span class="p">=</span> <span class="s2">"[</span><span class="p">$(</span><span class="nv">$ChoiceCharacter</span><span class="p">)</span><span class="s2">]</span><span class="p">$(</span><span class="nv">$Choice</span> <span class="o">-replace</span> <span class="s1">'_.'</span><span class="p">,</span><span class="s1">''</span><span class="p">)</span><span class="s2">"</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nv">$UserMenuItem</span> <span class="p">=</span> <span class="nv">$null</span>
<span class="k">While</span> <span class="p">(</span> <span class="o">-not</span> <span class="nv">$UserMenuItem</span> <span class="p">)</span> <span class="p">{</span>
<span class="no">[Console]</span><span class="p">::</span><span class="n">Clear</span><span class="p">()</span>
<span class="nb">Write-Host</span> <span class="s2">"------------------------"</span>
<span class="nb">Write-Host</span> <span class="s2">" My Rad Menu v0.1"</span>
<span class="nb">Write-Host</span> <span class="s2">"------------------------"</span>
<span class="c"># Display our menu</span>
<span class="k">Foreach</span> <span class="p">(</span> <span class="nv">$Item</span> <span class="k">in</span> <span class="nv">$MenuChoices</span> <span class="p">)</span> <span class="p">{</span>
<span class="nb">Write-Host</span> <span class="s2">" </span><span class="p">$(</span><span class="nv">$Item</span><span class="p">.</span><span class="n">MenuItem</span><span class="p">)</span><span class="s2">"</span>
<span class="p">}</span>
<span class="nb">Write-Host</span> <span class="s2">"------------------------"</span>
<span class="c"># Display a prompt and wait for user input</span>
<span class="nb">Write-Host</span> <span class="s2">"Pick an option: "</span> <span class="n">-NoNewline</span>
<span class="nv">$UserChoice</span> <span class="p">=</span> <span class="no">[Console]</span><span class="p">::</span><span class="n">ReadKey</span><span class="p">()</span>
<span class="c"># See if the user picked a valid item</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$UserChoice</span><span class="p">.</span><span class="n">KeyChar</span> <span class="n">-notin</span> <span class="nv">$MenuChoices</span><span class="p">.</span><span class="n">Key</span> <span class="p">)</span> <span class="p">{</span>
<span class="nb">Write-Host</span> <span class="s2">"</span><span class="se">`n</span><span class="s2">Invalid choice, try again. Press any key to continue..."</span>
<span class="nv">$null</span> <span class="p">=</span> <span class="no">[Console]</span><span class="p">::</span><span class="n">ReadKey</span><span class="p">(</span><span class="s1">'NoEcho'</span><span class="p">)</span>
<span class="p">}</span>
<span class="c"># Get the menu item that corresponds to the user choice</span>
<span class="nv">$UserMenuItem</span> <span class="p">=</span> <span class="nv">$MenuChoices</span> <span class="p">|</span> <span class="nb">Where-Object</span> <span class="p">{</span>
<span class="nv">$_</span><span class="p">.</span><span class="n">Key</span> <span class="o">-eq</span> <span class="nv">$UserChoice</span><span class="p">.</span><span class="n">Keychar</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c"># We display the users choice, inserting a newline to make up for the -NoNewLine above</span>
<span class="nb">Write-Host</span> <span class="s2">"</span><span class="se">`n</span><span class="s2">You chose </span><span class="p">$(</span><span class="nv">$UserMenuItem</span><span class="p">.</span><span class="n">MenuItem</span><span class="p">)</span><span class="s2">"</span>
</code></pre></div>
<p>This one is a bit more complicated, but let's see what's happening:</p>
<ul>
<li>Define our menu options</li>
<li>Clear the console <code>[Console]::Clear()</code></li>
<li>Loop through our menu options and display the menu text</li>
<li>Write a prompt to the screen</li>
<li>Call <code>[Console]::ReadKey()</code> which waits for user input</li>
<li>Check to see if the <code>KeyChar</code> property of the object returned by <code>ReadKey()</code> is a valid choice, if it isn't we display the menu and prompt again</li>
<li>Assuming the user picked a valid choice we just output the choice to the user</li>
</ul>
<blockquote>
<p>I am using a little RegEx magic here to take a menu choice like "_Continue" and break it into the character the user will specify to pick that option "C" (it will pick whatever character is after the underscore), and a menu item to display in the menu "[C]ontinue". If the choice was "Abort _Session", it would be displayed as "Abort [S]ession" and the 'S' would be used to select this menu item. This is a great way to create quick menus without a ton of code.</p>
</blockquote>
<p>This code could be extended to do almost anything you want. At the end we have the users choice and could easily build all sorts of logic around what to do next.</p>
<p>Now lets take this even further and introduce a the <code>KeyAvailable</code> property of <code>[Console]</code>. Like before, let's start with a simple example to see what it does:</p>
<div class="codehilite"><pre><span></span><code><span class="k">While</span> <span class="p">(</span> <span class="o">-not</span> <span class="no">[Console]</span><span class="p">::</span><span class="n">KeyAvailable</span> <span class="p">)</span> <span class="p">{</span>
<span class="nb">Start-Sleep</span> <span class="n">-Milliseconds</span> <span class="n">15</span>
<span class="p">}</span>
<span class="nb">Write-Host</span> <span class="s2">"A key was pressed!"</span>
</code></pre></div>
<p>This will sit in a loop until any key is pressed. It doesn't consume that keypress or do anything to process the keypress at all, it simply knows a key has been pressed and there is input in the input buffer that could be processed. </p>
<p>So what would we use this for? This is a fantastic tool when creating interactive monitoring scripts. Using this property we can sit in a loop, displaying changing information to user until they make a choice of some kind. Let's take a look at a simple example:</p>
<div class="codehilite"><pre><span></span><code><span class="k">While</span> <span class="p">(</span> <span class="o">-not</span> <span class="no">[Console]</span><span class="p">::</span><span class="n">KeyAvailable</span> <span class="p">)</span> <span class="p">{</span>
<span class="no">[Console]</span><span class="p">::</span><span class="n">Clear</span><span class="p">()</span>
<span class="nb">Get-Process</span> <span class="p">|</span> <span class="nb">Sort-Object</span> <span class="n">-Property</span> <span class="n">CPU</span> <span class="n">-Descending</span> <span class="p">|</span> <span class="nb">Select-Object</span> <span class="n">-First</span> <span class="n">10</span> <span class="p">|</span> <span class="nb">Out-String</span>
<span class="nb">Start-Sleep</span> <span class="n">-Milliseconds</span> <span class="n">2000</span>
<span class="p">}</span>
</code></pre></div>
<p>Here we are displaying the top 10 processes by CPU, and refreshing every 2 seconds until the user presses a key. We can combine this with our menu code above to create some very powerful scripts. For example, maybe we want this script to give the user an option to filter on process name, or alter the refresh period:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$Choices</span> <span class="p">=</span> <span class="p">@(</span>
<span class="s2">"_Filter Process Name"</span><span class="p">,</span>
<span class="s2">"_+ Increase Refresh Duration"</span>
<span class="s2">"_- Decrease Refresh Duration"</span>
<span class="s2">"E_xit"</span>
<span class="p">)</span>
<span class="nv">$MenuChoices</span> <span class="p">=</span> <span class="p">@()</span>
<span class="k">Foreach</span> <span class="p">(</span> <span class="nv">$Choice</span> <span class="k">in</span> <span class="nv">$Choices</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$ChoiceCharacter</span> <span class="p">=</span> <span class="nv">$Choice</span> <span class="o">-replace</span> <span class="s1">'.*_(.).*'</span><span class="p">,</span><span class="s1">'$1'</span>
<span class="nv">$MenuChoices</span> <span class="p">+=</span> <span class="p">@{</span>
<span class="s1">'Key'</span> <span class="p">=</span> <span class="nv">$ChoiceCharacter</span>
<span class="s1">'MenuItem'</span> <span class="p">=</span> <span class="nv">$Choice</span><span class="p">.</span><span class="n">Replace</span><span class="p">(</span><span class="s2">"_</span><span class="p">$(</span><span class="nv">$ChoiceCharacter</span><span class="p">)</span><span class="s2">"</span><span class="p">,</span><span class="s2">"[$ChoiceCharacter]"</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nv">$DurationMS</span> <span class="p">=</span> <span class="n">2000</span>
<span class="nv">$ProcessFilter</span> <span class="p">=</span> <span class="s2">""</span>
<span class="nv">$Run</span> <span class="p">=</span> <span class="nv">$true</span>
<span class="nv">$PParams</span> <span class="p">=</span> <span class="p">@{}</span>
<span class="k">While</span> <span class="p">(</span> <span class="nv">$Run</span> <span class="p">)</span> <span class="p">{</span>
<span class="c"># Clear the console</span>
<span class="no">[Console]</span><span class="p">::</span><span class="n">Clear</span><span class="p">()</span>
<span class="c"># Display process info</span>
<span class="nb">Get-Process</span> <span class="nv">@PParams</span> <span class="p">|</span> <span class="nb">Sort-Object</span> <span class="n">-Property</span> <span class="n">CPU</span> <span class="n">-Descending</span> <span class="p">|</span> <span class="nb">Select-Object</span> <span class="n">-First</span> <span class="n">10</span> <span class="p">|</span> <span class="nb">Out-String</span>
<span class="c"># Display a single-line prompt</span>
<span class="nb">Write-Host</span> <span class="s2">"</span><span class="p">$(</span><span class="nv">$MenuChoices</span><span class="p">.</span><span class="n">MenuItem-join</span><span class="p">(</span><span class="s1">', '</span><span class="p">))</span><span class="s2"> [</span><span class="p">$(</span><span class="nv">$DurationMS</span><span class="p">)</span><span class="s2">ms]: "</span> <span class="n">-NoNewline</span>
<span class="c"># Now lets sit in a loop and wait for input until the refresh duration passes or the user picks something</span>
<span class="nv">$StartWait</span> <span class="p">=</span> <span class="nb">Get-Date</span>
<span class="nv">$Wait</span> <span class="p">=</span> <span class="nv">$true</span>
<span class="k">While</span> <span class="p">(</span> <span class="p">((</span><span class="nb">Get-Date</span><span class="p">)</span> <span class="p">-</span> <span class="nv">$StartWait</span><span class="p">).</span><span class="n">TotalMilliseconds</span> <span class="o">-le</span> <span class="nv">$DurationMS</span> <span class="o">-and</span> <span class="nv">$Wait</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span> <span class="no">[Console]</span><span class="p">::</span><span class="n">KeyAvailable</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$UserChoice</span> <span class="p">=</span> <span class="no">[Console]</span><span class="p">::</span><span class="n">ReadKey</span><span class="p">()</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$UserChoice</span><span class="p">.</span><span class="n">KeyChar</span> <span class="n">-in</span> <span class="nv">$MenuChoices</span><span class="p">.</span><span class="n">Key</span> <span class="p">)</span> <span class="p">{</span>
<span class="k">switch</span> <span class="p">(</span><span class="nv">$UserChoice</span><span class="p">.</span><span class="n">KeyChar</span><span class="p">)</span> <span class="p">{</span>
<span class="n">F</span> <span class="p">{</span>
<span class="c"># Get the process to filter by</span>
<span class="nb">Write-Host</span> <span class="s2">"</span><span class="se">`n</span><span class="s2">Process Filter: "</span>
<span class="nv">$PFilter</span> <span class="p">=</span> <span class="no">[Console]</span><span class="p">::</span><span class="n">ReadLine</span><span class="p">()</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$PFilter</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$PParams</span><span class="p">.</span><span class="n">Name</span> <span class="p">=</span> <span class="s2">"*</span><span class="p">$(</span><span class="nv">$PFilter</span><span class="p">)</span><span class="s2">*"</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="c"># Remove the name filter if the choice is blank</span>
<span class="nv">$PParams</span><span class="p">.</span><span class="n">Remove</span><span class="p">(</span><span class="s2">"Name"</span><span class="p">)</span>
<span class="p">}</span>
<span class="nv">$Wait</span> <span class="p">=</span> <span class="nv">$False</span>
<span class="p">}</span>
<span class="p">-</span> <span class="p">{</span>
<span class="nv">$DurationMS</span> <span class="p">-=</span> <span class="n">500</span>
<span class="nv">$Wait</span> <span class="p">=</span> <span class="nv">$False</span>
<span class="p">}</span>
<span class="p">+</span> <span class="p">{</span>
<span class="nv">$DurationMS</span> <span class="p">+=</span> <span class="n">500</span>
<span class="nv">$Wait</span> <span class="p">=</span> <span class="nv">$False</span>
<span class="p">}</span>
<span class="n">X</span> <span class="p">{</span>
<span class="nv">$Wait</span> <span class="p">=</span> <span class="nv">$False</span>
<span class="nv">$Run</span> <span class="p">=</span> <span class="nv">$False</span>
<span class="nb">Write-Host</span> <span class="s2">"Exitting..."</span> <span class="n">-ForegroundColor</span> <span class="n">Red</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nb">Write-Host</span> <span class="s2">"Invalid Choice!"</span>
<span class="nb">Start-Sleep</span> <span class="n">-Milliseconds</span> <span class="n">500</span>
<span class="c"># Break the loop</span>
<span class="nv">$Wait</span> <span class="p">=</span> <span class="nv">$False</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nb">Start-Sleep</span> <span class="n">-Milliseconds</span> <span class="n">10</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>This example is a lot to go through line by line, but it just combines all the things we've already discussed. In general though, when this is executed:</p>
<ul>
<li>Enter a loop waiting for the state of <code>$Run</code> to change to false</li>
<li>Displays a list of current processes sorted by CPU</li>
<li>Displays a prompt for user input</li>
<li>Sits in a loop checking for user input every 10ms. It will loop until the refresh duration (2000ms by default) is reached or the <code>$Wait</code> variable is reset to <code>$False</code>, then it will get a list of processes again</li>
<li>When a key press is detected it will run through a <code>switch</code> statement and process the choice</li>
</ul>
<blockquote>
<p>TIP: When writing scripts like this, make sure you draw the output, then sit in a tight loop listening for user input. This makes the script much more responsive. If we instead waited for the full refresh duration, <code>$DurationMS</code> above, the script would appear slow and unresponsive</p>
</blockquote>
<p>Give this script a try and attempt to add your own changes and new options, it's the best way to learn!</p>
<h1>Conclusion</h1>
<p>Most folks think of PowerShell as a simple automation language, but you can do a LOT more if you use a little imagination. While I tend to shy away from using .Net objects/methods in my PowerShell code in favor of the native cmdlets, you can't deny the power and flexibility of .Net for the use case of user input. Thanks for reading!</p>
<p><strong>NOTE:</strong> Watch this space, as I will be releasing a SQL Server process monitor (similar to a sp_WhoIsActive) using the methods discussed in this post in the coming months. </p>associating vmware drives with guest os drives in powershell2020-11-05vmware_disk_mapHow to associate drives in vcenter with the corresponding disks in Windows.<p>Sometimes I get frustrated with how something is done and I put my head down and get to work. In this case it's working with VMWare disks.</p>
<p>On occasion we have to expand our virtual disks due to normal data growth. To do this we have to make a request to our System Operations team (they are a great team BTW and super responsive), in that request we have to include the exact size of the disk we need to expand because there is no easy way to tell which disk is which from inside the guest VM. This is where my frustration comes in. So I got to work and a few iterations of code later I found a solution.</p>
<h1>The Common Thread</h1>
<p>I must have poured over objects running <code>Get-Member</code> for hours trying to find the common thread that tied the Windows disk to the VMWare disk. In the end it was pretty simple: "physical" order</p>
<p>When configuring disks in VCenter you can configure one or more SCSI controllers:</p>
<p><img alt="List of SCSI controllers in VCenter" src="/img/20201102_Controllers.png" /></p>
<p>These are numbered starting with 0. If you look at the disks themselves, you will see they are identified by the controller id + the disk id for that controller:</p>
<p><img alt="List of disks in VCenter" src="/img/20201102_Disks.png" /></p>
<p>So how do the disks look in Windows? Well, you might think that the disk number in Windows would honor the order the disks are attached tot he virtual controllers, but you would be wrong. Just like I was. To see the disks in order of disk number, run the following code:</p>
<div class="codehilite"><pre><span></span><code><span class="nb">Invoke-Command</span> <span class="n">-ComputerName</span> <span class="n">server1</span><span class="p">.</span><span class="n">markw</span><span class="p">.</span><span class="n">dev</span> <span class="n">-ScriptBlock</span> <span class="p">{</span>
<span class="nb">Get-PhysicalDisk</span> <span class="p">|</span> <span class="nb">Sort-Object</span> <span class="n">-Property</span> <span class="n">DeviceID</span>
<span class="p">}</span> <span class="p">`</span>
<span class="p">|</span> <span class="nb">Select-Object</span> <span class="n">-Property</span> <span class="p">`</span>
<span class="n">DeviceID</span><span class="p">,</span>
<span class="p">@{</span><span class="n">Name</span><span class="p">=</span><span class="s2">"Size"</span><span class="p">;</span><span class="n">Expression</span><span class="p">={</span><span class="s2">"</span><span class="p">$(</span><span class="nv">$_</span><span class="p">.</span><span class="n">Size</span><span class="p">/</span><span class="n">1GB</span><span class="p">)</span><span class="s2">GB"</span><span class="p">}}</span>
</code></pre></div>
<p>In my example case I see the following: </p>
<div class="codehilite"><pre><span></span><code>DeviceID Size
-------- ----
0 50GB
1 10GB
2 50GB
3 70GB
4 32GB
5 25GB
6 75GB
</code></pre></div>
<p>If you refer to the screenshots from VCenter you'll see the order of the disks does NOT match. So what next? Well, we'll fast forward, but next I did a lot of work poking around in Windows and trying various commands in PowerShell piped into <code>Get-Member</code>. What I eventually found was pretty simple, the <code>PhysicalLocation</code> and <code>DeviceID</code> properties:</p>
<div class="codehilite"><pre><span></span><code><span class="nb">Invoke-Command</span> <span class="n">-ComputerName</span> <span class="n">server1</span><span class="p">.</span><span class="n">markw</span><span class="p">.</span><span class="n">dev</span> <span class="n">-ScriptBlock</span> <span class="p">{</span>
<span class="nb">Get-PhysicalDisk</span> <span class="p">|</span> <span class="nb">Sort-Object</span> <span class="n">-Property</span> <span class="n">DeviceID</span>
<span class="p">}</span> <span class="p">`</span>
<span class="p">|</span> <span class="nb">Select-Object</span> <span class="n">-Property</span> <span class="p">`</span>
<span class="n">PhysicalLocation</span><span class="p">,</span>
<span class="n">DeviceID</span><span class="p">,</span>
<span class="p">@{</span><span class="n">Name</span><span class="p">=</span><span class="s2">"Size"</span><span class="p">;</span><span class="n">Expression</span><span class="p">={</span><span class="s2">"</span><span class="p">$(</span><span class="nv">$_</span><span class="p">.</span><span class="n">Size</span><span class="p">/</span><span class="n">1GB</span><span class="p">)</span><span class="s2">GB"</span><span class="p">}}</span> <span class="n">-Unique</span> <span class="p">`</span>
<span class="p">|</span> <span class="nb">Sort-Object</span> <span class="n">-Property</span> <span class="p">`</span>
<span class="n">PhysicalLocation</span><span class="p">,</span>
<span class="n">DeviceID</span>
</code></pre></div>
<p>This gave me the following output:</p>
<div class="codehilite"><pre><span></span><code>PhysicalLocation DeviceID Size
---------------- -------- ----
SCSI0 0 50GB
SCSI0 1 10GB
SCSI0 2 50GB
SCSI1 5 25GB
SCSI1 6 75GB
SCSI2 3 70GB
SCSI2 4 32GB
</code></pre></div>
<p>If we again refer back to our screenshot, this disk order matches! </p>
<p>Let's put it all together now (<strong>NOTE</strong>: This function requires the VMWare PowerCLI PowerShell modules):</p>
<div class="codehilite"><pre><span></span><code><span class="k">function</span> <span class="nb">Get-VMDiskInformation</span> <span class="p">{</span>
<span class="cm"><#</span>
<span class="cm"> </span><span class="sd">.SYNOPSIS</span><span class="cm"></span>
<span class="cm"> Returns information about the VMWare volumes and associated</span>
<span class="cm"> Windows drives on a given guest.</span>
<span class="cm"> </span><span class="sd">.DESCRIPTION</span><span class="cm"></span>
<span class="cm"> Function connects to vSphere and a given Windows guest and</span>
<span class="cm"> returns the Windows phyical disk information mapped to the</span>
<span class="cm"> disks in VCenter.</span>
<span class="cm"> </span><span class="sd">.PARAMETER</span><span class="cm"> VSphereServer</span>
<span class="cm"> FQDN of vSphereServer that hosts the VM</span>
<span class="cm"> </span><span class="sd">.PARAMETER</span><span class="cm"> ComputerFQDN</span>
<span class="cm"> FQDN of the target server</span>
<span class="cm"> </span><span class="sd">.PARAMETER</span><span class="cm"> Credentials</span>
<span class="cm"> An option parameter to take a credential object in. This</span>
<span class="cm"> is then used to authenticate with VSphere</span>
<span class="cm"> #></span>
<span class="p">[</span><span class="k">CmdletBinding</span><span class="p">()]</span>
<span class="k">param</span>
<span class="p">(</span>
<span class="p">[</span><span class="k">Parameter</span><span class="p">(</span><span class="k">Mandatory</span><span class="p">=</span><span class="nv">$True</span><span class="p">)][</span><span class="n">ValidateNotNullOrEmpty</span><span class="p">()]</span>
<span class="no">[string]</span><span class="nv">$VSphereServer</span><span class="p">,</span>
<span class="p">[</span><span class="k">Parameter</span><span class="p">(</span><span class="k">Mandatory</span><span class="p">=</span><span class="nv">$True</span><span class="p">)][</span><span class="n">ValidateNotNullOrEmpty</span><span class="p">()]</span>
<span class="no">[string]</span><span class="nv">$ComputerFQDN</span><span class="p">,</span>
<span class="nv">$Credentials</span>
<span class="p">)</span>
<span class="nb">Import-Module</span> <span class="n">VMware</span><span class="p">.</span><span class="n">VimAutomation</span><span class="p">.</span><span class="n">Core</span>
<span class="c"># get user credentials to connect to vSphere</span>
<span class="k">if</span> <span class="p">(</span> <span class="o">-not</span> <span class="nv">$Credentials</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$Credentials</span> <span class="p">=</span> <span class="p">$(</span><span class="nb">Get-Credential</span><span class="p">)</span>
<span class="p">}</span>
<span class="c"># initialise arrays</span>
<span class="nv">$g_mounts</span> <span class="p">=</span> <span class="p">@()</span>
<span class="nv">$results</span> <span class="p">=</span> <span class="p">@()</span>
<span class="c"># set error action</span>
<span class="nv">$ErrorActionPreference</span> <span class="p">=</span> <span class="s2">"Stop"</span>
<span class="c"># connecting to vSphere</span>
<span class="nv">$Start</span> <span class="p">=</span> <span class="nb">Get-Date</span>
<span class="nb">Write-Host</span> <span class="s2">"Connection to VSphere Server..."</span>
<span class="nv">$vSphere</span> <span class="p">=</span> <span class="nb">Get-VIServer</span> <span class="n">-Server</span> <span class="nv">$VSphereServer</span> <span class="n">-Credential</span> <span class="nv">$Credentials</span> <span class="n">-Force</span>
<span class="c"># retrieving VM object</span>
<span class="nb">Write-Host</span> <span class="s2">"Getting VM Object..."</span>
<span class="nv">$VMName</span> <span class="p">=</span> <span class="nb">Get-View</span> <span class="n">-Viewtype</span> <span class="n">VirtualMachine</span> <span class="n">-Property</span> <span class="n">guest</span><span class="p">.</span><span class="n">hostname</span><span class="p">,</span> <span class="n">name</span> <span class="p">`</span>
<span class="p">|</span> <span class="nb">Where-Object</span> <span class="p">{</span> <span class="nv">$_</span><span class="p">.</span><span class="n">guest</span><span class="p">.</span><span class="n">hostname</span> <span class="o">-eq</span> <span class="nv">$ComputerFQDN</span> <span class="p">}</span> <span class="p">`</span>
<span class="p">|</span> <span class="nb">Select-Object</span> <span class="n">-ExpandProperty</span> <span class="n">Name</span>
<span class="nv">$VM</span> <span class="p">=</span> <span class="nb">Get-VM</span> <span class="n">-Server</span> <span class="nv">$vSphere</span><span class="p">.</span><span class="n">Name</span> <span class="n">-Name</span> <span class="nv">$VMName</span>
<span class="nb">Write-Host</span> <span class="s2">"Getting Disks..."</span>
<span class="c"># getting disk information</span>
<span class="nv">$v_disks</span> <span class="p">=</span> <span class="nv">$VM</span> <span class="p">|</span> <span class="nb">Get-HardDisk</span> <span class="p">|</span> <span class="nb">Sort-Object</span> <span class="n">-Property</span> <span class="n">Id</span>
<span class="nv">$Session</span> <span class="p">=</span> <span class="nb">New-CimSession</span> <span class="n">-ComputerName</span> <span class="nv">$ComputerFQDN</span>
<span class="c"># Some servers return duplicate entries when grabbing the physical disks,</span>
<span class="c"># so we first grab the in-use serial numbers to filter the physical disks</span>
<span class="nv">$g_disk_serials</span> <span class="p">=</span> <span class="nb">Get-Disk</span> <span class="n">-CimSession</span> <span class="nv">$Session</span> <span class="p">|</span> <span class="nb">Select-Object</span> <span class="n">-ExpandProperty</span> <span class="n">SerialNumber</span>
<span class="c"># get physical disk info, but only for disks with serial numbers that appear</span>
<span class="c"># in `Get-Disk`. We NEED the physical location to properly map the drives.</span>
<span class="nv">$g_disks</span> <span class="p">=</span> <span class="nb">Get-PhysicalDisk</span> <span class="n">-CimSession</span> <span class="nv">$Session</span> <span class="p">`</span>
<span class="p">|</span> <span class="nb">Where-Object</span> <span class="p">{</span> <span class="nv">$_</span><span class="p">.</span><span class="n">SerialNumber</span> <span class="n">-in</span> <span class="nv">$g_disk_serials</span> <span class="p">}</span> <span class="p">`</span>
<span class="p">|</span> <span class="nb">Select-Object</span> <span class="n">-Property</span> <span class="n">PhysicalLocation</span><span class="p">,</span><span class="n">DeviceID</span><span class="p">,</span><span class="n">SerialNumber</span> <span class="n">-Unique</span> <span class="p">`</span>
<span class="p">|</span> <span class="nb">Sort-Object</span> <span class="n">-Property</span> <span class="n">PhysicalLocation</span><span class="p">,</span><span class="n">DeviceID</span>
<span class="nb">Write-Verbose</span> <span class="s2">"Disk Info:"</span>
<span class="nb">Write-Verbose</span> <span class="s2">" - VMWare Disks: </span><span class="p">$(</span><span class="nv">$v_disks</span><span class="p">.</span><span class="n">Count</span><span class="p">)</span><span class="s2">"</span>
<span class="nv">$v_disks</span> <span class="p">|</span> <span class="k">ForEach</span><span class="n">-Object</span> <span class="p">{</span>
<span class="nb">Write-Verbose</span> <span class="s2">" - </span><span class="p">$(</span><span class="nv">$_</span><span class="p">)</span><span class="s2">"</span>
<span class="p">}</span>
<span class="nb">Write-Verbose</span> <span class="s2">" - Guest Disks: </span><span class="p">$(</span><span class="nv">$g_disks</span><span class="p">.</span><span class="n">Count</span><span class="p">)</span><span class="s2">"</span>
<span class="nv">$g_disks</span> <span class="p">|</span> <span class="k">ForEach</span><span class="n">-Object</span> <span class="p">{</span>
<span class="nb">Write-Verbose</span> <span class="s2">" - </span><span class="p">$(</span><span class="nv">$_</span><span class="p">)</span><span class="s2">"</span>
<span class="p">}</span>
<span class="nb">Write-Host</span> <span class="s2">"Finding mount points..."</span>
<span class="c"># adding disk information to g_mounts array, ignore empty mount paths</span>
<span class="nv">$CimPartInfo</span> <span class="p">=</span> <span class="nb">Get-Partition</span> <span class="n">-CimSession</span> <span class="nv">$Session</span> <span class="p">`</span>
<span class="p">|</span> <span class="nb">Where-Object</span> <span class="p">{</span> <span class="nv">$_</span><span class="p">.</span><span class="n">AccessPaths</span> <span class="p">}</span> <span class="p">`</span>
<span class="p">|</span> <span class="nb">Select-Object</span> <span class="n">-Property</span> <span class="n">DiskNumber</span><span class="p">,</span><span class="n">AccessPaths</span><span class="p">,</span><span class="n">PartitionNumber</span>
<span class="k">foreach</span> <span class="p">(</span><span class="nv">$CimPart</span> <span class="k">in</span> <span class="p">(</span><span class="nv">$CimPartInfo</span> <span class="p">|</span> <span class="nb">Where-Object</span> <span class="p">{</span> <span class="nv">$_</span><span class="p">.</span><span class="n">AccessPaths</span> <span class="p">}</span> <span class="p">))</span> <span class="p">{</span>
<span class="nv">$g_mounts</span> <span class="p">+=</span> <span class="no">[PSCustomObject]</span><span class="p">@{</span>
<span class="n">Path</span> <span class="p">=</span> <span class="nv">$CimPart</span><span class="p">.</span><span class="n">AccessPaths</span><span class="p">[</span><span class="n">0</span><span class="p">]</span>
<span class="n">DiskNumber</span> <span class="p">=</span> <span class="nv">$CimPart</span><span class="p">.</span><span class="n">DiskNumber</span>
<span class="n">PartitionNumber</span> <span class="p">=</span> <span class="nv">$CimPart</span><span class="p">.</span><span class="n">PartitionNumber</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nb">Write-Verbose</span> <span class="s2">"Mount Point Info:"</span>
<span class="nv">$g_mounts</span> <span class="p">|</span> <span class="k">ForEach</span><span class="n">-Object</span> <span class="p">{</span>
<span class="nb">Write-Verbose</span> <span class="s2">" - ID: </span><span class="p">$(</span><span class="nv">$_</span><span class="p">.</span><span class="n">DiskNumber</span><span class="p">)</span><span class="s2"> - </span><span class="p">$(</span><span class="nv">$_</span><span class="p">.</span><span class="n">Path</span><span class="p">)</span><span class="s2">"</span>
<span class="p">}</span>
<span class="c"># map the vm disks to the guest disks (include mount points if used)</span>
<span class="k">for</span><span class="p">(</span><span class="nv">$i</span><span class="p">=</span><span class="n">0</span><span class="p">;</span><span class="nv">$i</span> <span class="o">-lt</span> <span class="nv">$v_disks</span><span class="p">.</span><span class="n">Count</span><span class="p">;</span><span class="nv">$i</span><span class="p">++){</span>
<span class="nv">$v_disk</span> <span class="p">=</span> <span class="nv">$v_disks</span><span class="p">[</span><span class="nv">$i</span><span class="p">]</span>
<span class="nv">$g_disk</span> <span class="p">=</span> <span class="nv">$g_disks</span><span class="p">[</span><span class="nv">$i</span><span class="p">]</span>
<span class="nv">$g_mount</span> <span class="p">=</span> <span class="nv">$g_mounts</span> <span class="p">|</span> <span class="nb">Where-Object</span> <span class="p">{</span> <span class="nv">$_</span><span class="p">.</span><span class="n">DiskNumber</span> <span class="o">-eq</span> <span class="nv">$g_disk</span><span class="p">.</span><span class="n">DeviceId</span> <span class="o">-and</span> <span class="nv">$_</span><span class="p">.</span><span class="n">Path</span> <span class="p">}</span>
<span class="c"># adding disk information to results array</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$g_drives</span> <span class="o">-notcontains</span> <span class="n">900</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$results</span> <span class="p">+=</span> <span class="no">[PSCustomObject]</span><span class="p">@{</span>
<span class="c"># In cases where a disk has multiple partitions and multiple mount points</span>
<span class="c"># we only care about reporting the disk number once.</span>
<span class="n">DiskNumber</span> <span class="p">=</span> <span class="nv">$g_mount</span><span class="p">.</span><span class="n">DiskNumber</span> <span class="p">|</span> <span class="nb">Select </span><span class="n">-First</span> <span class="n">1</span>
<span class="n">PartitionNumber</span> <span class="p">=</span> <span class="nv">$g_mount</span><span class="p">.</span><span class="n">PartitionNumber</span>
<span class="n">VMWareDisk</span> <span class="p">=</span> <span class="nv">$v_disk</span><span class="p">.</span><span class="n">Name</span>
<span class="n">DiskSerial</span> <span class="p">=</span> <span class="nv">$g_disk</span><span class="p">.</span><span class="n">SerialNumber</span>
<span class="n">Path</span> <span class="p">=</span> <span class="nv">$g_mount</span><span class="p">.</span><span class="n">Path</span>
<span class="n">CapacityGB</span> <span class="p">=</span> <span class="nv">$v_disk</span><span class="p">.</span><span class="n">CapacityGB</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c"># return results array</span>
<span class="nv">$results</span>
<span class="p">}</span>
</code></pre></div>
<p>[<a href="/public/GetVMDiskInformation.ps1.txt">download this script</a>]</p>
<p>This function does a little more than we covered above, but to keep it simple it does the following:</p>
<ul>
<li>Gets the virtual disk associated with a given guest</li>
<li>Gets the physical disks according to Windows for the given guest</li>
<li>Uses the <code>PhysicalLocation</code> and <code>DeviceID</code> properties to match the disks up with their VMWare counterparts</li>
</ul>
<p>This function goes a step farther as well and tells you which mountpoint the disk is mounted on (if mountpoints are used).</p>
<h1>Conclusion</h1>
<p>This project was a LOT harder than it should have been. I did a lot of searching and never really found a simple way to do this. Most of the posts I came across recommended using the disk size to do the matching but this just felt really hacky to me. In the final version of this function I also access data from a Pure Storage array to identify the specific VVol associated with the guest disk, but that seemed a little to specific for a blog post. If you are interested in that part, send me an <a href="/about" title="about me">email</a>.</p>eightkb is back!2020-11-17eightkb2021EightKB - The SQL Server Internals conference - is back in 2021!<p>EightKB in July 2020 was a shocking success with over 1,000 concurrent attendees at our peak! What started as a few comments in a private Slack turned into one of the biggest virtual SQL Server events of 2020! Since then 2020 has been a hot mess so I am super excited to announce that EightKB 2021 registration is <strong>OPEN</strong>!</p>
<h1><a href="https://eightkb.online">REGISTER NOW!</a></h1>
<p>Join us on January 27th 2021 @ 14:00 UTC for a day of brain-melting technical content, fantastic speakers, and the awesome SQL Server community everyone knows and loves. Just like last time this is a 100% <strong>FREE</strong> event run for and by the community! No sponsors, no vendors, just content and community.</p>
<h2>The lineup</h2>
<table>
<thead>
<tr>
<th>Time</th>
<th>Speaker</th>
<th align="center">Length</th>
<th align="center">Level</th>
<th>Title</th>
</tr>
</thead>
<tbody>
<tr>
<td>14:15</td>
<td>Amit Bansal</td>
<td align="center">75m</td>
<td align="center">400</td>
<td><a href="https://eightkb.online/detail.html?session_id=196958">SQL Memory Internals and Troubleshooting</a></td>
</tr>
<tr>
<td>15:45</td>
<td>Bob Pusateri</td>
<td align="center">60m</td>
<td align="center">400</td>
<td><a href="https://eightkb.online/detail.html?session_id=197214">The Ins and Outs of SQL Server Data Compression</a></td>
</tr>
<tr>
<td>17:00</td>
<td>Gail Shaw</td>
<td align="center">75m</td>
<td align="center">400</td>
<td><a href="https://eightkb.online/detail.html?session_id=200577">Intelligent Query Processing, what's up with that?</a></td>
</tr>
<tr>
<td>18:30</td>
<td>Klaus Aschenbrenner</td>
<td align="center">75m</td>
<td align="center">400</td>
<td><a href="https://eightkb.online/detail.html?session_id=197458">Latches, spinlocks, and lock free data structures</a></td>
</tr>
<tr>
<td>20:00</td>
<td>Thomas Grohser</td>
<td align="center">75m</td>
<td align="center">400</td>
<td><a href="https://eightkb.online/detail.html?session_id=197215">Scaling SQL Server beyond 2 CPUs</a></td>
</tr>
</tbody>
</table>
<h2>How can I support this amazing event?</h2>
<p>Tell your friends, co-workers, and followers! You can also help support us by stopping in at the <a href="https://bonfire.com/store/eightkb">EightKB store</a> and picking up a Mixed Extents t-shirt (EightKB shirts soon to come)!</p>
<h2>Can't wait?</h2>
<p>If you can't wait for Janurary check out our last event on our <a href="https://www.youtube.com/playlist?list=PLr9ab4Dj3ObsGA8jdstJFZfidQ51DvPvu">YouTube Channel</a> and subscribe to our podcast on <a href="https://www.youtube.com/playlist?list=PLr9ab4Dj3ObsOZyOfjSMXJmL11zZnyYoO">YouTube</a> or <a href="https://eightkb.online/mixedextents/">wherever fine podcasts are sold</a> (just kidding, it's free). </p>2020 in review2020-12-232020_reviewLooking back at this messed up year.<p>As 2020 comes to a close I had a desire to write a blog post about all the things and feelings I've had over this year. It's been a hard year for folks, and I'll admit that as a white male in tech, I've had it much easier than others. For me this year has been long stretches of time where the days and weeks blended and blurred together all punctuated by memorable events and experiences. For others it's meant lost jobs and lost loved ones, so I do feel fortunate.</p>
<h1>Pandemic Family</h1>
<p>The first thing I want to talk about is my "pandemic family", Andrew Pruski (<a href="https://dbafromthecold.com">B</a>|<a href="https://twitter.com/dbafromthecold">T</a>) and Anthony Nocentino (<a href="https://nocentinosystems.com/blog">B</a>|<a href="https://twitter.com/nocentino">T</a>). Without these two guys 2020 would have been a hell of a lot harder than it was. Even with Zoom fatigue, we were always available if anyone needed to hop on a chat about something (or about nothing). It was great having some people to lean on whenever I needed it. So big thanks goes out to both of them. </p>
<h1>EightKB and Mixed Extents</h1>
<p><a href="https://eightkb.online">EightKB</a> and <a href="https://eightkb.online/mixedextents">Mixed Extents</a> have been a blast to organize and take part in. While I am admittedly ready for in-person events to start back up, having virtual events to fill the time has been great. The time involved is a good distraction and I've gotten to meet a lot of folks in the community I've never had a chance to talk to before. I've also gotten to fulfill a long-time dream of mine: organizing a dedicated internals conference. I'm excited to see how events shape up in the future and of course look forward to future EightKB events and episodes of Mixed Extents.</p>
<h1>Diversity and Inclusion</h1>
<p>As I said at the start, I'm a white male in tech, my life in tech is not hard. I hate to say that prior to this year I didn't really "get" diversity and inclusion. I thought of myself as a "non-racist" person, and someone that would give anyone a chance regardless of gender/race/sexual orientation/religion. This year I learned that it's not enough to just give people a fair chance, I need to use my position of privilege to make sure others are not overlooked. This is especially important as an event organizer and hiring manager. Making sure underrepresented groups get fair and equal access to all the opportunities others enjoy <em>has</em> to be a priority if we ever want anything to change.</p>
<h1>PASS</h1>
<p>What would a year in review be without talking about PASS? I've struggled to write a post about PASS, I know there are a lot feelings and opinions floating around out there and I of course have some of my own. I started my time with PASS in 2013-ish when I start attending a local user group in Detroit Michigan. I had no idea what PASS was at the time, but had heard it mentioned. From there I attended my first SQL Saturday (and spoke at it) which infected me with the speaking bug.</p>
<p>Since then I have spoken at several events, ran SQL Clinics, and helped others get up the nerve to start speaking. It's been a lot of fun but I've never had a really strong attachment to PASS itself, I think I came in at the wrong time. So when it started looking like PASS might not survive the year it didn't really hit me that hard. I knew the community would live on and the events would as well. I do have concerns about another central organization emerging and starting the cycle over again, but I have hope that the community will build something better this time around.</p>
<h1>Looking Forward</h1>
<p>I'm under no delusions that 2021 is going to magically be a better year. I think it has potential though. With the COVID vaccine being administered to folks as I type this, and a new administration coming into the White House, I think we can expect better things to come. At a community level I think 2020 will have some hidden benefits that we'll see for years to come. This year has seen lots of new events and ideas, even if some of the old ones have gone away. To end this in true 2020
fashion I'll leave you with a quote from Jeff Goldblum's character from <a href="https://youtu.be/oijEsqT2QKQ">Jurassic Park</a>:</p>
<blockquote>
<p>No, I'm simply saying that life, uh, finds a way. <br> - Dr. Ian Malcolm</p>
</blockquote>query parsing bug in sql server2021-02-01parse_bugAn interesting SQL Server parsing bug we found in the wild.<p>In this post I'll discuss a bug we found in SQL Server in our staging environment. This is an interesting bug that seems to have come into existence when <code>CREATE OR ALTER</code> support was added to SQL Server. When you create a new procedure and use <code>CREATE OR ALTER</code>, SQL Server removes the <code>OR ALTER</code> in the create statement before storing the text of the procedure. You can confirm this yourself:</p>
<div class="codehilite"><pre><span></span><code><span class="k">CREATE</span><span class="w"> </span><span class="k">OR</span><span class="w"> </span><span class="k">ALTER</span><span class="w"> </span><span class="k">PROCEDURE</span><span class="w"> </span><span class="n">dbo</span><span class="p">.</span><span class="n">TestProc</span><span class="w"></span>
<span class="k">AS</span><span class="w"></span>
<span class="k">SELECT</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">EXEC</span><span class="w"> </span><span class="n">sp_helptext</span><span class="w"> </span><span class="s1">'dbo.TestProc'</span><span class="p">;</span><span class="w"></span>
</code></pre></div>
<p>You'll notice that <code>OR ALTER</code> has been removed in the definition of the procedure. </p>
<h1>The bug</h1>
<p>So now on to the good stuff, why you are all here. Where I work, we add comments to the top of all of our procedures (above the <code>CREATE</code>) to add notes about changes that were made. Recently we had a procedure that compiled fine, but would not execute. It reported a syntax error, which is odd since it compiled fine:</p>
<div class="codehilite"><pre><span></span><code>Msg 156, Level 15, State 1, Procedure dbo.test, Line 46
Incorrect syntax near the keyword 'ALTER'.
Msg 102, Level 15, State 1, Procedure dbo.test, Line 50
Incorrect syntax near 'SELEC'.
</code></pre></div>
<p>When we inspected the procedure text, we saw something odd:</p>
<div class="codehilite"><pre><span></span><code><span class="k">CREATE</span><span class="w"> </span><span class="k">ALTER</span><span class="w"> </span><span class="k">PROCEDURE</span><span class="w"> </span><span class="n">dbo</span><span class="p">.</span><span class="n">TestProc</span><span class="w"></span>
<span class="k">AS</span><span class="w"></span>
<span class="n">SELE</span><span class="w"></span>
</code></pre></div>
<p>So what's happening? </p>
<h1>Troubleshooting</h1>
<p>The first thing we thought is that maybe there was a special character in the comments above the <code>CREATE</code> that were causing trouble. We deleted a single-quote and the proc compiled and executed fine! Cool, problem solved. At that point we noticed the proc was also missing a newline we usually insert after the comment block. We added it and found that the procedure would no longer execute! WHAT IS HAPPENING? So after a bit of fiddling around adding and removing characters (including unicode just for fun) we discovered the issue:</p>
<p><strong>If you have 3,988 BYTES of data, you hit some sort of boundary bug that, instead of clipping the 5 characters "ALTER" from the query text, clips the last 5 characters from the query text!</strong></p>
<h1>The repro</h1>
<p>Reproducing this bug is fairly straight forward, you just execute this script: <a href="/public/test.sql">test.sql</a></p>
<p>Here is the text of the script (do not copy/paste as it may introduce extra characters):</p>
<div class="codehilite"><pre><span></span><code><span class="cm">/*</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA</span>
<span class="cm">*/</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">OR</span><span class="w"> </span><span class="k">ALTER</span><span class="w"> </span><span class="k">PROCEDURE</span><span class="w"> </span><span class="p">[</span><span class="n">dbo</span><span class="p">].[</span><span class="n">test</span><span class="p">]</span><span class="w"></span>
<span class="k">AS</span><span class="w"></span>
<span class="k">SELECT</span><span class="w"> </span><span class="mi">1</span><span class="w"></span>
</code></pre></div>
<p>And that's it! When you attempt you execute the procedure you'll get the error from above!</p>
<h1>Whats the big deal?</h1>
<p>So why is this a big deal? Originally we thought it could lead to cases where part of a where clause would get clipped off, but since it also sees the "ALTER" statement as a syntax error, it would never get to the WHERE clause. Where this could be a big deal is if you deploy your changes to test environments via SSMS, and to your production environments via some sort of automation. In that case the procedure will execute fine until it hits production. There you will be met with a syntax error that makes no sense, and no real idea how it got in that state or how to fix it.</p>
<h1>How do we fix this?</h1>
<p>No idea. I currently have a support case in with Microsoft and they are working on reproducing the issue. I think since the bug relates to <code>CREATE OR ALTER</code> it should be pretty easy to figure out where in the code the bug exists. This has been tested in everything from 2017 CU15 up to the latest version of 2019 (in a container) and it remains broken in all the versions I tested. I would be interested to hear if others have run into this, or are unable to reproduce it on their own systems.</p>self-hosting presentations using reveal2021-02-22revealUsing reveal.js and your existing web hosting to host your presentations.<p>In a <a href="https://dbafromthecold.com/2021/02/21/creating-presentations-with-reveal-and-github-pages/">recent blog post</a> Andrew Pruski (<a href="https://dbafromthecold.com/">b</a>|<a href="https://twitter.com/dbafromthecold/">t</a>) shared a presentation setup he put together using Github Pages and reveal.js. He had been using the soon-to-be defunct <a href="https://gitpitch.com">GitPitch</a> for his presentations, and needed something new. In this post I'll share the setup I shared with Andrew that allows me to self-host my reveal presentations. I like what Andrew is using, but my own setup fits my needs and lets me host my presentations the same way I host my blog, so I thought I would share. With this setup all I have to do to create a new presentation is create a new <code>.md</code> file and optionally update an existing file.</p>
<h1>Reveal.js</h1>
<p>First off, if you haven't used <a href="https://revealjs.com/">Reveal.js</a> before, you should check it out. Reveal allows you to write your presentations using <a href="https://daringfireball.net/projects/markdown/syntax">Markdown</a> (a simple to use, human-readable, mark-up language) and host them almost anywhere (even locally using a python one-liner).</p>
<p>Reveal supports a lot of great features like syntax highlighting and speaker notes. It makes writing presentations a breeze and makes you hate PowerPoint more than you already do.</p>
<h1>The setup</h1>
<p>Before we can get started you need two things:</p>
<ul>
<li>A directory on your hosting setup that you can host your presentations out of</li>
<li>A copy of reveal.js</li>
</ul>
<p>I created a directory called <code>/presentations</code> in the root of my hosting directory. In that directory I have a <code>js/</code> directory, an <code>index.html</code> file, and a file named <code>null.md</code>. </p>
<ul>
<li><code>js/</code> - A copy of reveal.js lives here</li>
<li><code>index.html</code> - A skeleton file that just loads a chosen presentation</li>
<li><code>null.md</code> - The "default" presentation that shows when you browse to the <code>presentations/</code> directory</li>
</ul>
<p>To get the latest copy of reveal.js, change into your <code>presentations/js/</code> directory and run:</p>
<div class="codehilite"><pre><span></span><code>git clone https://github.com/hakimel/reveal.js.git
</code></pre></div>
<blockquote>
<p>You only need the <code>js</code>,<code>dist</code>, and <code>plugin</code> directories in that repo, the rest can be deleted.</p>
</blockquote>
<h2>Index.html</h2>
<p>Now that you have reveal.js installed, you need an <code>index.html</code> file to server the presentations. Here is mine:</p>
<div class="codehilite"><pre><span></span><code><span class="cp"><!doctype html></span>
<span class="p"><</span><span class="nt">html</span><span class="p">></span>
<span class="p"><</span><span class="nt">head</span><span class="p">></span>
<span class="p"><</span><span class="nt">meta</span> <span class="na">charset</span><span class="o">=</span><span class="s">"utf-8"</span><span class="p">></span>
<span class="p"><</span><span class="nt">meta</span> <span class="na">name</span><span class="o">=</span><span class="s">"viewport"</span> <span class="na">content</span><span class="o">=</span><span class="s">"width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no"</span><span class="p">></span>
<span class="p"><</span><span class="nt">title</span><span class="p">></span>reveal.js<span class="p"></</span><span class="nt">title</span><span class="p">></span>
<span class="p"><</span><span class="nt">link</span> <span class="na">rel</span><span class="o">=</span><span class="s">"stylesheet"</span> <span class="na">href</span><span class="o">=</span><span class="s">"js/reveal.js/dist/reset.css"</span><span class="p">></span>
<span class="p"><</span><span class="nt">link</span> <span class="na">rel</span><span class="o">=</span><span class="s">"stylesheet"</span> <span class="na">href</span><span class="o">=</span><span class="s">"js/reveal.js/dist/reveal.css"</span><span class="p">></span>
<span class="p"><</span><span class="nt">link</span> <span class="na">rel</span><span class="o">=</span><span class="s">"stylesheet"</span> <span class="na">href</span><span class="o">=</span><span class="s">"js/reveal.js/dist/theme/black.css"</span> <span class="na">id</span><span class="o">=</span><span class="s">"theme"</span><span class="p">></span>
<span class="cm"><!-- Theme used for syntax highlighted code --></span>
<span class="p"><</span><span class="nt">link</span> <span class="na">rel</span><span class="o">=</span><span class="s">"stylesheet"</span> <span class="na">href</span><span class="o">=</span><span class="s">"js/reveal.js/plugin/highlight/monokai.css"</span> <span class="na">id</span><span class="o">=</span><span class="s">"highlight-theme"</span><span class="p">></span>
<span class="p"></</span><span class="nt">head</span><span class="p">></span>
<span class="p"><</span><span class="nt">body</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"reveal"</span><span class="p">></span>
<span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"slides"</span><span class="p">></span>
<span class="p"><</span><span class="nt">section</span> <span class="na">name</span><span class="o">=</span><span class="s">"md_content"</span>
<span class="na">data-markdown</span><span class="o">=</span><span class="s">"slides.md"</span>
<span class="na">data-separator</span><span class="o">=</span><span class="s">"^---$"</span>
<span class="na">data-separator-vertical</span><span class="o">=</span><span class="s">"^------$"</span>
<span class="na">data-separator-notes</span><span class="o">=</span><span class="s">"^Note:"</span>
<span class="na">data-charset</span><span class="o">=</span><span class="s">"iso-8859-15"</span><span class="p">></span>
><span class="p"></</span><span class="nt">section</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"></</span><span class="nt">div</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">"js/reveal.js/dist/reveal.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">"js/reveal.js/plugin/notes/notes.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">"js/reveal.js/plugin/markdown/markdown.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span> <span class="na">src</span><span class="o">=</span><span class="s">"js/reveal.js/plugin/highlight/highlight.js"</span><span class="p">></</span><span class="nt">script</span><span class="p">></span>
<span class="p"><</span><span class="nt">script</span><span class="p">></span>
<span class="kd">var</span> <span class="nx">md</span> <span class="o">=</span> <span class="nb">document</span><span class="p">.</span><span class="nx">getElementsByName</span><span class="p">(</span><span class="s1">'md_content'</span><span class="p">)[</span><span class="mf">0</span><span class="p">]</span>
<span class="kd">var</span> <span class="nx">params</span> <span class="o">=</span> <span class="ow">new</span> <span class="nx">URLSearchParams</span><span class="p">(</span><span class="nb">window</span><span class="p">.</span><span class="nx">location</span><span class="p">.</span><span class="nx">search</span><span class="p">)</span>
<span class="nx">md</span><span class="p">.</span><span class="nx">dataset</span><span class="p">.</span><span class="nx">markdown</span> <span class="o">=</span> <span class="s2">"/presentations/"</span> <span class="o">+</span> <span class="nx">params</span><span class="p">.</span><span class="nx">get</span><span class="p">(</span><span class="s1">'p'</span><span class="p">)</span> <span class="o">+</span> <span class="s2">".md"</span>
<span class="c1">// More info about initialization & config:</span>
<span class="c1">// - https://revealjs.com/initialization/</span>
<span class="c1">// - https://revealjs.com/config/</span>
<span class="nx">Reveal</span><span class="p">.</span><span class="nx">initialize</span><span class="p">({</span>
<span class="nx">hash</span><span class="o">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="c1">// Learn about plugins: https://revealjs.com/plugins/</span>
<span class="nx">plugins</span><span class="o">:</span> <span class="p">[</span> <span class="nx">RevealMarkdown</span><span class="p">,</span> <span class="nx">RevealHighlight</span><span class="p">,</span> <span class="nx">RevealNotes</span> <span class="p">]</span>
<span class="p">});</span>
<span class="p"></</span><span class="nt">script</span><span class="p">></span>
<span class="p"></</span><span class="nt">body</span><span class="p">></span>
<span class="p"></</span><span class="nt">html</span><span class="p">></span>
</code></pre></div>
<p>Nothing really special is happening here. It imports all the various reveal libraries, a style sheet for syntax highlighting, and one for the presentations as a whole. We then do a little setup for reveal, and finally we do the magic part: </p>
<div class="codehilite"><pre><span></span><code>var md = document.getElementsByName('md_content')[0]
var params = new URLSearchParams(window.location.search)
md.dataset.markdown = "/presentations/" + params.get('p') + ".md"
</code></pre></div>
<p>The first line of code gets finds <code>md_content</code> element (this is where the presentation content will go). Next it grabs the value of the <code>p</code> URL parameter, and finally we use that value to pull information from a markdown file and render the presentation. </p>
<p>As an example, if I browsed to <a href="https://markw.dev/presentations/?p=hello">https://markw.dev/presentations/?p=hello</a>, the code would see the <code>?p=hello</code> in the URL and then find the <code>hello.md</code> in the presentations directory. It would then parse the markdown in that file and render the presentation. This is cool because you can just load that directory up with <code>.md</code> files and change the URL parameter to switch between presentations!</p>
<h2>null.md</h2>
<p>What about <code>null.md</code> though, I mentioned it earlier, but what is it for? If the <code>p</code> URL parameter is missing, the js code above will return <code>null</code>, so if I create <code>null.md</code> file, it will be displayed when a parameter isn't entered. I took advantage of this behavior and created links to my various presentations in that file:</p>
<div class="codehilite"><pre><span></span><code>### Please choose a presentation:
* [Monitoring and Alerting](/presentations/?p=alerting)
* [SQL Server CPU Scheduling](/presentations/?p=cpu)
* [Untangling Dynamic SQL](/presentations/?p=dynamic)
</code></pre></div>
<p>So, if you just browse to <a href="https://markw.dev/presentations/">https://markw.dev/presentations/</a> without specifying <code>p</code>, it will show you a list of my available presentations. This is an option step of course, you could just just as easily create a file with a help message and not bother maintaining a list of your presentations:</p>
<div class="codehilite"><pre><span></span><code># No presentation selected!
Use the `?p=` url parameter to choose a presentation
</code></pre></div>
<h1>Thoughts</h1>
<p>I will admit that my way is a little more complicated to get up and running, but I like being able to just drop a new <code>.md</code> file in that directory and I am ready to go. Local testing is dead simple as well. All I have to do is change to the <code>presnetations/</code> directory locally and use a python one-liner to serve it on a local web server:</p>
<div class="codehilite"><pre><span></span><code><span class="n">python3</span> <span class="o">-</span><span class="n">m</span> <span class="n">http</span><span class="o">.</span><span class="n">server</span> <span class="mi">8080</span>
</code></pre></div>
<p>That serves the current directory on <code>http://localhost:8080</code> and lets me quickly test changes before uploading it. Reveal is super flexible and results in a great looking presentation that is extremely easy to share (there is even a PDF printing option). That combined with the ease of creating presentations in markdown, and the ease of putting those presentations in source control, should really make anyone think twice about using PowerPoint ever again.</p>combining sendto and powershell2021-03-01sendto_powershellHow to execute PowerShell scripts via the Windows 'SendTo' menu.<p>Recently Jess Pomfret (<a href="https://jesspomfret.com/">b</a>|<a href="https://twitter.com/jpomfret">t</a>) wrote a <a href="https://jesspomfret.com/execute-folder-of-scripts/">post</a> on executing a folder of SQL scripts against a SQL Server instance. It's a great post I recommend checking out, and it reminded me of something I have been meaning to blog about for a while: creating custom <em>SendTo</em> options in Windows that execute PowerShell scripts.</p>
<h1>SendTo</h1>
<p><img alt="Send To Menu" src="/img/sendto_menu.png" />
If you are not familiar, SendTo options are those available when you right click on a file/folder in file explorer and select the <em>Send To</em> option in the menu. When you use this option, the currently selected files/folders are passed to the SendTo shortcut as a space delimited list of files and folders. This is important to know so you better understand what needs to be done to read that list.</p>
<h1>The PowerShell</h1>
<p>The PowerShell code is pretty straight forward, the script will ask the user for a parameter value (this could be used with Jess's example to enter a server name), and then simply list the files and pause for the user to press Enter. This script could be customized to do almost anything you need: </p>
<div class="codehilite"><pre><span></span><code><span class="k">param</span><span class="p">(</span>
<span class="p">[</span><span class="k">Parameter</span><span class="p">(</span><span class="k">ValueFromRemainingArguments</span><span class="p">=</span><span class="nv">$true</span><span class="p">)]</span>
<span class="nv">$FilePath</span>
<span class="p">)</span>
<span class="k">while</span> <span class="p">(</span> <span class="o">-not</span> <span class="nv">$MandatoryParameter</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$MandatoryParameter</span> <span class="p">=</span> <span class="nb">Read-Host</span> <span class="s2">"Enter this Mandatory Parameter"</span>
<span class="p">}</span>
<span class="c"># Expand folders into scripts</span>
<span class="nb">Get-ChildItem</span> <span class="n">-Recurse</span> <span class="n">-Path</span> <span class="nv">$FilePath</span> <span class="p">|</span> <span class="k">ForEach</span><span class="n">-Object</span> <span class="p">{</span>
<span class="nb">Write-Host</span> <span class="s2">" - </span><span class="p">$(</span><span class="nv">$_</span><span class="p">.</span><span class="n">FullName</span><span class="p">)</span><span class="s2">"</span>
<span class="p">}</span>
<span class="n">Pause</span>
</code></pre></div>
<p>There are a few important things to know about using scripts in this way. First, you see <code>ValueFromRemainingArguments</code> above? That is how you handle the space delimited list of files passed in when you use "Send To". Second, if you are used to the typical behavior of adding <code>Mandatory</code> parameters that prompt the user for input, that doesn't work here. In this case I like to just use a while loop that loops until the variable is set. If this were a production script I would also do some validation in that loop to make sure the user supplied value makes sense. </p>
<h1>Creating the Shortcut</h1>
<p>While not required, I find it easiest to put the above code in a script file directly in the <code>SendTo</code> directory. You can get to that directory by either opening file explorer and typing <code>SendTo</code> in the address bar, or by browsing to <code>%APPDATA%\Microsoft\Windows\SendTo</code>. Just putting the script in that directory isn't enough though, we also have to create a new shortcut. To create the shortcut browse to the <code>SendTo</code> directory, right-click, and select <em>New > Shortcut</em>. This will open the new shortcut wizard. </p>
<p>The first thing it asks for the location of the command. Unfortunately you can't just browse to the path of the script and expect it to work, you also have to add the powershell.exe path to the command. Assuming a script named <code>SendToTest.ps1</code> exists you would enter the following:</p>
<div class="codehilite"><pre><span></span><code>C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -file %APPDATA%\Microsoft\Windows\SendTo\TestSendTo.ps1
</code></pre></div>
<p>Next it asks for the name, this is what will show up in your <code>Send To</code> menu, so make it good! In this case I will just use <code>SendTo Test</code>.</p>
<blockquote>
<p>If you have a set of static parameters you would like to pass to your scripts, you can either set default values right in the script, or supply them as parameters when creating the shortcut. For example, adding <code>-Parameter "MyValue"</code> to the command line above.</p>
</blockquote>
<h1>Using it</h1>
<p>Now for the moment of truth. Select a group of scripts and try out the new shortcut:
<img alt="File Explorer Right-Click Menu" src="/img/sendto_files.png" /></p>
<p>If it all worked you should see output similar to this:</p>
<div class="codehilite"><pre><span></span><code>Enter this Mandatory Parameter: TestValue
- C:\Users\mark\test_scripts\4_TestScript.sql
- C:\Users\mark\test_scripts\5_TestScript.sql
- C:\Users\mark\test_scripts\6_TestScript.sql
- C:\Users\mark\test_scripts\7_TestScript.sql
- C:\Users\mark\test_scripts\8_TestScript.sql
- C:\Users\mark\test_scripts\9_TestScript.sql
- C:\Users\mark\test_scripts\10_TestScript.sql
- C:\Users\mark\test_scripts\1_TestScript.sql
- C:\Users\mark\test_scripts\2_TestScript.sql
- C:\Users\mark\test_scripts\3_TestScript.sql
Press Enter to continue...:
</code></pre></div>
<blockquote>
<p>If you didn't see this glorious output, and instead a window opened and then quickly closed, you likely have to adjust the execution policy. I personally set mine to <code>bypass</code>, but make sure to read up on this option before changing it. If you are confident in how I have my machines configured, run this to get around this issue: <code>Set-ExecutionPolicy -ExecutionPolicy Bypass -Scope LocalMachine -Force</code></p>
</blockquote>
<h1>Thoughts</h1>
<p>The coolest thing about this is that it's all just standard PowerShell. If you are using it for deploying a set of scripts you could integrate code to look up a server list via a CMS server, you could introduce runspaces to do things in parallel, you could even tie it into a ticketing system to report when scripts were deployed. The main point thought is that it is a great, low friction, way to start setting up automation for common tasks.</p>hosting email for close to free on aws ses2021-05-12aws-free-emailHow to setup up email for your domain using AWS SES, a few lines of Python, and a local dovecot install.<p>Anyone who knows me knows how cheap I am, especially when it comes to paying for services I feel I can run myself. Email is no exception. I've used Gmail since it was invite-only beta, but I've always wanted to have email that was untethered from the big email hosting companies out there. In the past I attempted to run a mail server through a terrible VPS provider and though I got it all working, I was burned when the VPS hosting company suddenly went under in the middle of the night.</p>
<p>This brings us to the present. When I started this blog, I wanted to host my own email in some fashion without just forwarding my domain specific email to a Gmail address. In this post we are going to go over a setup I put together over the course of a few hours that allows me to host mail in the cloud nearly for free.</p>
<h1>Overview</h1>
<p>This setup is surprisingly simple yet very flexible. To get this up and running you'll need to set up the following:</p>
<ul>
<li>AWS Simple Email Service (SES)</li>
<li>AWS S3 Bucket</li>
<li>Linux Server with Python3 and Dovecot (an IMAP server) installed</li>
<li>A TLS certificate (LetsEncrypt is a great option)</li>
</ul>
<p>The Linux server in this setup can be hosted in the cloud or a home server. Most ISPs don't block the ports required to run an IMAP server at home (you could also use a non-standard port). Many ISPs also have extremely long leases on IP addresses. If you want something more stable though, a cloud hosted instance might be a better option. That being said, AWS does provide a free-tier EC2 option that should be able to handle a Dovecot install.</p>
<h1>AWS SES</h1>
<p>AWS SES is designed for programatically sending marketing blasts and transactional email. There are a few interesting features that make it work well for personal email as well. Setting up SES is pretty simple:</p>
<ul>
<li>Navigate to Simple Email Service in the AWS Console</li>
<li>The first thing you need to do is verify that you own the domain you want to configure email for, this is done under <em>Identity Mangement</em></li>
<li><strong>TIP</strong> Verify the domain! All through the documentation they really push for validating specific "From" addresses. If you just verify the whole domain, you don't have to validate specific addresses</li>
<li>To start the process, click on "Verify a New Domain"</li>
<li>In the window that follows, make sure to check the box to generate DKIM settings</li>
<li>Click on <em>Verify This Domain</em> to continue</li>
<li>You'll be presented with a screen that lists a number of DNS records you need to add to your domain.</li>
<li>After adding these records, your domain should eventually show as <strong>verified</strong> under both <em>Verification Status</em> and <em>DKIM Status</em></li>
</ul>
<p>At this point, your domain is ready to send email. In the next section we'll go through what needs to happen to set up your first email address.</p>
<p>Before we get an email account set up, we also need to configure SMTP via SES. This is what allows you to send mail from your domain via an email client. To get this configured:</p>
<ul>
<li>Click on <em>SMTP Settings</em> in the SES Dashboard</li>
<li>Click on <em>Create My SMTP Credentials</em> and follow the prompts</li>
</ul>
<p>At the end, it will give you the details you need to connect your mail client to SES.</p>
<blockquote>
<p><strong>TIP</strong> For further compatibility, I would recommend setting up a custom MAIL FROM domain. You can configure this setting via the <em>Domains</em> option from the SES Dashboard. In my case I set up a sub-domain of <strong>mail</strong> and used it as my FROM domain. When you configure this it will give you a new MX record to add to your domain as well as an SPF record that is used by email servers to validate that your email came from the correct domain.</p>
</blockquote>
<h1>AWS SES to S3</h1>
<p>One option you have with SES is to send all incoming mail to a specific S3 bucket. This is a super flexible option. Once the files are in S3, you can process them any way you see fit. In my bucket I have a <code>processed</code> directory where I move mail that I have already processed on my local IMAP server.</p>
<p>Check out the AWS Documentation on setting up SES to forward to an S3 bucket <a href="https://aws.amazon.com/premiumsupport/knowledge-center/ses-receive-inbound-emails/">here</a>.</p>
<blockquote>
<p><strong>TIP</strong> Set up a lifecycle policy on your S3 bucket to remove files older than 30 days. This will keep costs down and will also speed up mail retrieval by the Python script later in this post.</p>
</blockquote>
<h1>Dovecot</h1>
<p>Next, we will set up an IMAP server to serve our mail for consumption by an email client. This is pretty dead simple. I do recommend getting a TLS cert for your domain. The best way to do this is to use the DNS verification method of the Let's Encrypt cert generation process. You can read more about that <a href="https://letsencrypt.org/how-it-works/">here</a>. In order for this cert to work you'll also need to make sure to set up a DNS A record for your home server. I personally created an A Record to point the <code>imap</code> subdomain to my home IP address and created the cert for that subdomain as well.</p>
<p>I am not going to cover how to install Dovecot — it should be in your distribution's package manager — but I will provide my configuration file:</p>
<div class="codehilite"><pre><span></span><code><span class="n">mail_home</span><span class="o">=/</span><span class="k">var</span><span class="o">/</span><span class="n">mail</span><span class="o">/%</span><span class="n">n</span><span class="w"></span>
<span class="n">mail_location</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mbox</span><span class="p">:</span><span class="o">~/</span><span class="n">mail</span><span class="p">:</span><span class="n">INBOX</span><span class="o">=/</span><span class="k">var</span><span class="o">/</span><span class="n">mail</span><span class="o">/%</span><span class="n">u</span><span class="w"></span>
<span class="c1"># if you want to use system users</span><span class="w"></span>
<span class="n">passdb</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="n">driver</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">pam</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">userdb</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="n">driver</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">passwd</span><span class="w"></span>
<span class="n">args</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">blocking</span><span class="o">=</span><span class="n">no</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">ssl</span><span class="o">=</span><span class="n">yes</span><span class="w"></span>
<span class="n">ssl_cert</span><span class="o">=</</span><span class="n">etc</span><span class="o">/</span><span class="n">dovecot</span><span class="o">/</span><span class="n">ssl</span><span class="o">/</span><span class="n">cert</span><span class="o">.</span><span class="n">pem</span><span class="w"></span>
<span class="n">ssl_key</span><span class="o">=</</span><span class="n">etc</span><span class="o">/</span><span class="n">dovecot</span><span class="o">/</span><span class="n">ssl</span><span class="o">/</span><span class="n">privkey</span><span class="o">.</span><span class="n">pem</span><span class="w"></span>
<span class="c1"># if you are using v2.3.0-v2.3.2.1 (or want to support non-ECC DH algorithms)</span><span class="w"></span>
<span class="c1"># since v2.3.3 this setting has been made optional.</span><span class="w"></span>
<span class="c1">#ssl_dh=</path/to/dh.pem</span><span class="w"></span>
<span class="n">namespace</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="n">inbox</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">yes</span><span class="w"></span>
<span class="n">separator</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">/</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="n">protocols</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">imap</span><span class="w"> </span>
<span class="n">disable_plaintext_auth</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">yes</span><span class="w"></span>
<span class="n">mail_privileged_group</span><span class="o">=</span><span class="n">mail</span><span class="w"></span>
</code></pre></div>
<p>This configuration will give you a functional IMAP server that uses your local username/password for authentication. For example, if you have a local account named "mark" you would authenticate with your IMAP server with the same username and password the "mark" user would use to log in to a shell on that server.</p>
<p>This user does NOT need to match the email address you want to use with SES. In the next step, we will be pull email out of S3 and put it in the local server's mailbox for a user of our choice (via the <code>/var/mail/<user></code> mbox file).</p>
<h1>A Little Python</h1>
<p>At this point we have incoming mail hitting an S3 bucket and a functional local IMAP server; now what? With a little Python code, we are going to download these mail message files to our local server and parse them into a standard Linux MBOX file. The MBOX format is simple: messages are stored in one big file, with new messages appended to the end. You can read more about the MBOX format <a href="https://tools.ietf.org/rfc/rfc4155.txt">here</a>.</p>
<p>I have created a Python script that does everything I need to get these messages downloaded to my local server that does the following:</p>
<ul>
<li>Downloads any new files from S3</li>
<li>Parses the message and pulls out the sender and date the message was received</li>
<li>Builds a standard MBOX message delimiter line</li>
<li>Appends the message to the user MBOX file</li>
<li>Moves the S3 message file to the <code>processed/</code> directory</li>
</ul>
<p>The MBOX message delimiter line is pretty straightforward. It must start with the word <strong>From</strong>, followed by the sender name, email address, and the utc date/time the message was received in Unix ctime format (for example: <code>Fri 07 Feb 2020 20:24:40</code>)</p>
<p>The script is fairly simple:</p>
<div class="codehilite"><pre><span></span><code><span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">boto3</span>
<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="kn">from</span> <span class="nn">email.parser</span> <span class="kn">import</span> <span class="n">Parser</span>
<span class="kn">from</span> <span class="nn">email.policy</span> <span class="kn">import</span> <span class="n">default</span>
<span class="n">mail_box</span> <span class="o">=</span> <span class="s1">'/var/mail/<user>'</span>
<span class="n">msg_dir</span> <span class="o">=</span> <span class="s1">'/tmp/'</span>
<span class="n">s3_bucket</span> <span class="o">=</span> <span class="s1">'<s3-bucket-name>'</span>
<span class="c1"># Create s3 client</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">boto3</span><span class="o">.</span><span class="n">client</span><span class="p">(</span><span class="s1">'s3'</span><span class="p">)</span>
<span class="c1"># Get new mail</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">list_objects_v2</span><span class="p">(</span>
<span class="n">Bucket</span><span class="o">=</span><span class="n">s3_bucket</span><span class="p">,</span>
<span class="n">Delimiter</span><span class="o">=</span><span class="s1">','</span>
<span class="p">)</span>
<span class="n">new_messages</span> <span class="o">=</span> <span class="p">[</span><span class="n">msg</span><span class="p">[</span><span class="s1">'Key'</span><span class="p">]</span> <span class="k">for</span> <span class="n">msg</span> <span class="ow">in</span> <span class="n">response</span><span class="p">[</span><span class="s1">'Contents'</span><span class="p">]</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">msg</span><span class="p">[</span><span class="s1">'Key'</span><span class="p">]</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'processed/'</span><span class="p">)]</span>
<span class="k">if</span> <span class="n">new_messages</span><span class="p">:</span>
<span class="n">s3</span> <span class="o">=</span> <span class="n">boto3</span><span class="o">.</span><span class="n">resource</span><span class="p">(</span><span class="s1">'s3'</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">new_messages</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">message_content</span> <span class="o">=</span> <span class="n">s3</span><span class="o">.</span><span class="n">Object</span><span class="p">(</span><span class="n">s3_bucket</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()[</span><span class="s1">'Body'</span><span class="p">]</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">UnicodeDecodeError</span><span class="p">:</span>
<span class="n">message_content</span> <span class="o">=</span> <span class="n">s3</span><span class="o">.</span><span class="n">Object</span><span class="p">(</span><span class="n">s3_bucket</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">()[</span><span class="s1">'Body'</span><span class="p">]</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'cp1252'</span><span class="p">)</span>
<span class="c1"># If the e-mail headers are in a file, uncomment these two lines:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">content</span> <span class="o">=</span> <span class="n">Parser</span><span class="p">(</span><span class="n">policy</span><span class="o">=</span><span class="n">default</span><span class="p">)</span><span class="o">.</span><span class="n">parsestr</span><span class="p">(</span><span class="n">message_content</span><span class="p">)</span>
<span class="c1"># Or for parsing headers in a string (this is an uncommon operation), use:</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">mail_box</span><span class="p">,</span> <span class="s1">'a'</span><span class="p">)</span> <span class="k">as</span> <span class="n">mb</span><span class="p">:</span>
<span class="k">if</span> <span class="s1">'Date'</span> <span class="ow">in</span> <span class="n">content</span><span class="p">:</span>
<span class="n">header_date</span> <span class="o">=</span> <span class="n">content</span><span class="p">[</span><span class="s1">'Date'</span><span class="p">]</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">header_date</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">utcnow</span><span class="p">()</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s2">"</span><span class="si">%a</span><span class="s2"> </span><span class="si">%d</span><span class="s2"> %b %Y %H:%M:%S +0000"</span><span class="p">)</span>
<span class="n">from_addr</span> <span class="o">=</span> <span class="n">content</span><span class="p">[</span><span class="s1">'From'</span><span class="p">]</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">'"'</span><span class="p">,</span><span class="s1">''</span><span class="p">)</span>
<span class="n">header_string</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">"From </span><span class="se">\"</span><span class="si">{</span><span class="n">from_addr</span><span class="si">}</span><span class="se">\"</span><span class="s2"> </span><span class="si">{</span><span class="n">header_date</span><span class="si">}</span><span class="s2">"</span>
<span class="n">mb</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">header_string</span><span class="p">)</span>
<span class="n">mb</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">message_content</span><span class="p">)</span>
<span class="n">s3</span><span class="o">.</span><span class="n">Object</span><span class="p">(</span><span class="n">s3_bucket</span><span class="p">,</span><span class="sa">f</span><span class="s2">"processed/</span><span class="si">{</span><span class="n">i</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span><span class="o">.</span><span class="n">copy_from</span><span class="p">(</span><span class="n">CopySource</span><span class="o">=</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="n">s3_bucket</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">i</span><span class="si">}</span><span class="s2">"</span><span class="p">)</span>
<span class="n">s3</span><span class="o">.</span><span class="n">Object</span><span class="p">(</span><span class="n">s3_bucket</span><span class="p">,</span><span class="n">i</span><span class="p">)</span><span class="o">.</span><span class="n">delete</span><span class="p">()</span>
<span class="k">except</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">"Something went wrong!"</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">exc_info</span><span class="p">()[</span><span class="mi">0</span><span class="p">])</span>
</code></pre></div>
<p>I will admit this could be cleaned up a bit, but it's a v1 project. Fill in the <code>mail_box</code> and <code>s3_bucket</code> variables with the location of your MBOX file and the s3 bucket you created for SES. After that, you can schedule this to run in a cron job every minute and you should start seeing emails come through to your mail client.</p>
<h1>DMARC</h1>
<p>While not required, setting DMARC is a good way to make sure your mail is getting delivered and accepted by most mail servers. To set up DMARC, add a new DNS TXT record with a name of <code>_dmarc</code> and a value of <code>"v=DMARC1;p=quarantine;pct=25;ruf=mailto:<email address>"</code> replacing the email address with your new address. This will send you reports if your outgoing email is ever blocked by a destination mail server.</p>
<h1>Conclusion</h1>
<p>With this setup, when someone sends mail to your address, AWS SES receives it and stores the message in an S3 bucket. A Python script then pulls the message down from S3 and appends it to the local MBOX file. Finally, that message is served up via a local IMAP server. It looks complicated but if you have experience with Linux and AWS, it shouldn't be too difficult to set up.</p>
<p>I have been running this setup for over a year now and it's been working great. I had a few issues early on (and still find an occasional email that messes things up) when I discovered that not all incoming message headers are formatted consistently, but luckily I found out about the Python <code>email</code> module that handles this for me. I like that this gets me away from the big mail providers (I am not ignorant to the fact that Amazon is likely scanning all of my incoming mail) and that I am free to do what I want with the message files (like storing them off site, or putting them into a document store for better searching). While it's not for everyone, I am impressed with how much you can get done only using free tier services. </p>sqltop - an interactive sql server process viewer2021-05-20sqltopRead about my new tool, sqltop, and why I wrote it.<p>Hey folks! I'm proud to announce the first open source release of my <a href="https://github.com/channeladvisor/sqltop">sqltop</a> tool! <code>sqltop</code> is an interactive command-line based tool to view active sessions on a SQL Server instance. In this post I'll talk about why I wrote the tool, why I chose to write it in PowerShell, and walk through some of the challenges I faced during development.</p>
<p><img alt="sqltop main screen" src="/img/sqltop_screen.png" /></p>
<h1>Why?</h1>
<p>For those not familiar with the Linux command line, there is a fantastic tool called <code>top</code>, and a slightly more fantastic tool called <code>htop</code>. To over simplify, these tools allow you to see a real-time view of the processes running on the system. This includes the path to the executable, the user running the process, resources being consumed, etc. <code>sqltop</code> is an attempt to bring the same tools to SQL Server.</p>
<p>It's not meant to replace monitoring, or even the many well known community tools. It's just meant as another tool to have in your tool belt when trying to troubleshoot performance issues on busy servers. This need is highly driven by the environment I work in. If something goes wrong and work starts to pile up, a server can quickly get to the point where running things like <code>sp_who2</code> won't even return results. Because of this we needed something we could use in almost any situation that would give us all the info we would need to start triaging an issue. </p>
<h1>What's in the box?</h1>
<p>As you can see in the screenshot above, <code>sqltop</code> shows most of the basics you'd expect, as well as a few extra things like multiple lines if a session is running a parallel task, and an indicator if the session is taking part in a snapshot transaction.</p>
<p>One feature that sets this tool apart is the display refresh. The display refreshes every 5 seconds with new data from the server. If the server is under extreme load the data refresh process will slowly back off, increasing time between refreshes, until the server starts responding faster. The auto-refresh alone is a handy feature but you can also:</p>
<ul>
<li>Highlight sessions based on an arbitrary text search string</li>
<li>Capture plans and sql text for a specific session, or even kill a running session</li>
<li>Track a running session over time to watch as it accumulates waits and resource usage stats</li>
</ul>
<p>You can also switch between a number of views to see the data in a different way. For example you can see waits by session, cumulative waits for all sessions, resource usage by program or stored procedure call, and even a blocking tree view:</p>
<p><img alt="Screen showing the blocking tree for all blocked sessions" src="/img/sqltop_blocking.png" /></p>
<h1>Why PowerShell?</h1>
<p>Probably one of the first things folks ask me about this tool is "Why PowerShell?". Why indeed. PowerShell is 100% <strong>not</strong> the best language to write an interactive CLI app. That being said, my initial goal with <code>sqltop</code> was to get more experience working with PowerShell runspaces, specifically passing data around between them. I learn by doing so I grabbed a diagnostic query I had been working on and decided to put a quick UI around it. If I had known we would actually end up using it and eventually open-sourcing the project I might have written it in Go or Python. But here we are. </p>
<p>Using PowerShell for this project turned out to be very educational as I ran into a lot of strange challenges I had to overcome. For the remainder of this post I'll go over the more interesting challenges and what my ultimate solution was. </p>
<blockquote>
<p>Quick Note: Throughout this project I am sure there are places where external modules would have helped, but I wanted to keep it as lean and self-contained as possible. I spent a lot of time shaving milliseconds off here and there to make the UI a smooth experience, introducing external modules would likely not have made this task easier.</p>
</blockquote>
<h2>Sharing data and state</h2>
<p><code>sqltop</code> uses a separate runspace to retrieve information from SQL. This process runs separate from the UI loop. This model gave me a ton of flexibility as the UI is never blocked by the process that queries for session information. This allows for a smooth user experience as the UI can respond to commands almost immediately.</p>
<p>To accomplish this separation, I needed to figure out how best to pass query result data to the UI, and also have UI driven events change the state of the process running the queries and collecting data. What I ended up using a was a <code>Synchronized Hashtable</code>. </p>
<p>Creating a syncronized hashtable is pretty simple: </p>
<div class="codehilite"><pre><span></span><code><span class="nv">$StateData</span> <span class="p">=</span> <span class="no">[hashtable]</span><span class="p">::</span><span class="n">Synchronized</span><span class="p">(@{})</span>
</code></pre></div>
<p>This creates a threadsafe hashtable, allowing us to access this variable from both the session the UI code is running on, and the runspace code the data retrieval code is running on. </p>
<p>Next I had to create a new runspace and inject this variable into it:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$newRunspace</span> <span class="p">=</span> <span class="no">[runspacefactory]</span><span class="p">::</span><span class="n">CreateRunspace</span><span class="p">()</span>
<span class="nv">$newRunspace</span><span class="p">.</span><span class="n">Open</span><span class="p">()</span>
<span class="nv">$newRunspace</span><span class="p">.</span><span class="n">SessionStateProxy</span><span class="p">.</span><span class="n">SetVariable</span><span class="p">(</span><span class="s2">"StateData"</span><span class="p">,</span><span class="nv">$StateData</span><span class="p">)</span>
</code></pre></div>
<p>Now that we have <code>$StateData</code> available in both the current session and the runspace, we can do things like run a SQL query and store the results in <code>$StateData.Results</code>, making those results available to the UI. For some additional safety I also added a boolean property to the hashtable called <code>Lock</code>. This gets set to <code>$True</code> at the beginning of the UI loop and prevents the result data from being updated while the display is being drawn. </p>
<p>I learned a lot getting this portion of the code to work and ended up using quite a few special properties to send signals between the UI and data processes.</p>
<h2>Invoke-Sqlcmd</h2>
<p><code>Invoke-Sqlcmd</code> is typically your go-to when you need to query SQL Server. Sure it has it's odd behaviors, but overall it's reliable. This is not the case when you try to run <code>Invoke-Sqlcmd</code> from within a new runspace. Within a runspace I ran into a lot of odd error messages (which I have forgotten about as it's a been a while) and odd behavior in error cases. In this case the solution was simple, I just used .NET objects directly:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$Conn</span> <span class="p">=</span> <span class="no">[System.Data.Sqlclient.SqlConnection]</span><span class="p">::</span><span class="n">new</span><span class="p">()</span>
<span class="nv">$Conn</span><span class="p">.</span><span class="n">ConnectionString</span> <span class="p">=</span> <span class="nv">$ConnectionString</span>
<span class="nv">$Conn</span><span class="p">.</span><span class="n">Open</span><span class="p">()</span>
<span class="nv">$Cmd</span> <span class="p">=</span> <span class="no">[System.Data.Sqlclient.SqlCommand]</span><span class="p">::</span><span class="n">new</span><span class="p">()</span>
<span class="nv">$Cmd</span><span class="p">.</span><span class="n">CommandText</span> <span class="p">=</span> <span class="nv">$Query</span>
<span class="nv">$Cmd</span><span class="p">.</span><span class="n">CommandTimeout</span> <span class="p">=</span> <span class="n">108000</span>
<span class="nv">$Cmd</span><span class="p">.</span><span class="n">Connection</span> <span class="p">=</span> <span class="nv">$Conn</span>
<span class="nv">$SqlAdapter</span> <span class="p">=</span> <span class="nb">New-Object</span> <span class="n">System</span><span class="p">.</span><span class="n">Data</span><span class="p">.</span><span class="n">SqlClient</span><span class="p">.</span><span class="n">SqlDataAdapter</span>
<span class="nv">$SqlAdapter</span><span class="p">.</span><span class="n">SelectCommand</span> <span class="p">=</span> <span class="nv">$Cmd</span>
<span class="nv">$DataSet</span> <span class="p">=</span> <span class="nb">New-Object</span> <span class="n">System</span><span class="p">.</span><span class="n">Data</span><span class="p">.</span><span class="n">DataSet</span>
<span class="nv">$null</span> <span class="p">=</span> <span class="nv">$SqlAdapter</span><span class="p">.</span><span class="n">Fill</span><span class="p">(</span><span class="nv">$DataSet</span><span class="p">)</span>
<span class="nv">$Results</span> <span class="p">=</span> <span class="nv">$DataSet</span><span class="p">.</span><span class="n">Tables</span><span class="p">[</span><span class="n">0</span><span class="p">].</span><span class="n">Rows</span>
</code></pre></div>
<p>The query I am running is a known quantity so the code to execute it didn't really need to be that robust. I didn't put specific error handling code in either, as if the data retrieval code fails in any, it passes the error message directly to the UI to let the user know what's happening.</p>
<h2>Color!</h2>
<p>While this may not seem super important at first, color ended up being important to the interface and allowed for some interesting interactions. The biggest challenge related to multi-color display is doing things like highlighting specific words or lines in a multi-line output with a different color. For example, there is no way to output a table with alternating colored lines using with standard PowerShell functions. </p>
<p><img alt="Close-up of alternating color rows" src="/img/sqltop_altcolors.png" /></p>
<p>This was accomplished by converting the output (in normal powershell table format) into a string using <code>Out-String -Stream</code>. This converts what would have been displayed on the screen into a string, and <code>-Stream</code> emits it line-by-line. It is then piped through <code>ForEach-Object</code>. Following this pattern I can now count which line I am on an decide to make the line a different color (for alternating row colors) or I can search the line for a given search string and highlight the line. </p>
<p>Now that I have this line-by-line control I simply used color escape codes to color the output. Escape codes allow you to color down to the individual character so it gives you a lot of control and flexibility. The only issue is that it is a little verbose to use, so I wrote a small helper function called <code>color</code>:</p>
<div class="codehilite"><pre><span></span><code><span class="k">function</span> <span class="n">color</span> <span class="p">{</span>
<span class="k">param</span><span class="p">(</span>
<span class="nv">$Text</span><span class="p">,</span>
<span class="nv">$ForegroundColor</span> <span class="p">=</span> <span class="s1">'default'</span><span class="p">,</span>
<span class="nv">$BackgroundColor</span> <span class="p">=</span> <span class="s1">'default'</span>
<span class="p">)</span>
<span class="c"># Terminal Colors</span>
<span class="nv">$Colors</span> <span class="p">=</span> <span class="p">@{</span>
<span class="s2">"default"</span> <span class="p">=</span> <span class="p">@(</span><span class="n">40</span><span class="p">,</span><span class="n">50</span><span class="p">)</span>
<span class="s2">"black"</span> <span class="p">=</span> <span class="p">@(</span><span class="n">30</span><span class="p">,</span><span class="n">0</span><span class="p">)</span>
<span class="s2">"lightgrey"</span> <span class="p">=</span> <span class="p">@(</span><span class="n">33</span><span class="p">,</span><span class="n">43</span><span class="p">)</span>
<span class="s2">"grey"</span> <span class="p">=</span> <span class="p">@(</span><span class="n">37</span><span class="p">,</span><span class="n">47</span><span class="p">)</span>
<span class="s2">"darkgrey"</span> <span class="p">=</span> <span class="p">@(</span><span class="n">90</span><span class="p">,</span><span class="n">100</span><span class="p">)</span>
<span class="s2">"red"</span> <span class="p">=</span> <span class="p">@(</span><span class="n">91</span><span class="p">,</span><span class="n">101</span><span class="p">)</span>
<span class="s2">"darkred"</span> <span class="p">=</span> <span class="p">@(</span><span class="n">31</span><span class="p">,</span><span class="n">41</span><span class="p">)</span>
<span class="s2">"green"</span> <span class="p">=</span> <span class="p">@(</span><span class="n">92</span><span class="p">,</span><span class="n">102</span><span class="p">)</span>
<span class="s2">"darkgreen"</span> <span class="p">=</span> <span class="p">@(</span><span class="n">32</span><span class="p">,</span><span class="n">42</span><span class="p">)</span>
<span class="s2">"yellow"</span> <span class="p">=</span> <span class="p">@(</span><span class="n">93</span><span class="p">,</span><span class="n">103</span><span class="p">)</span>
<span class="s2">"white"</span> <span class="p">=</span> <span class="p">@(</span><span class="n">97</span><span class="p">,</span><span class="n">107</span><span class="p">)</span>
<span class="s2">"brightblue"</span> <span class="p">=</span> <span class="p">@(</span><span class="n">94</span><span class="p">,</span><span class="n">104</span><span class="p">)</span>
<span class="s2">"darkblue"</span> <span class="p">=</span> <span class="p">@(</span><span class="n">34</span><span class="p">,</span><span class="n">44</span><span class="p">)</span>
<span class="s2">"indigo"</span> <span class="p">=</span> <span class="p">@(</span><span class="n">35</span><span class="p">,</span><span class="n">45</span><span class="p">)</span>
<span class="s2">"cyan"</span> <span class="p">=</span> <span class="p">@(</span><span class="n">96</span><span class="p">,</span><span class="n">106</span><span class="p">)</span>
<span class="s2">"darkcyan"</span> <span class="p">=</span> <span class="p">@(</span><span class="n">36</span><span class="p">,</span><span class="n">46</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$ForegroundColor</span> <span class="n">-notin</span> <span class="nv">$Colors</span><span class="p">.</span><span class="n">Keys</span> <span class="o">-or</span> <span class="nv">$BackgroundColor</span> <span class="n">-notin</span> <span class="nv">$Colors</span><span class="p">.</span><span class="n">Keys</span><span class="p">)</span> <span class="p">{</span>
<span class="nb">Write-Error</span> <span class="s2">"Invalid color choice!"</span> <span class="n">-ErrorAction</span> <span class="n">Stop</span>
<span class="p">}</span>
<span class="s2">"</span><span class="p">$(</span><span class="no">[char]</span><span class="n">27</span><span class="p">)</span><span class="s2">[</span><span class="p">$(</span><span class="nv">$colors</span><span class="p">[</span><span class="nv">$ForegroundColor</span><span class="p">][</span><span class="n">0</span><span class="p">])</span><span class="s2">m</span><span class="p">$(</span><span class="no">[char]</span><span class="n">27</span><span class="p">)</span><span class="s2">[</span><span class="p">$(</span><span class="nv">$colors</span><span class="p">[</span><span class="nv">$BackgroundColor</span><span class="p">][</span><span class="n">1</span><span class="p">])</span><span class="s2">m</span><span class="p">$(</span><span class="nv">$Text</span><span class="p">)$(</span><span class="no">[char]</span><span class="n">27</span><span class="p">)</span><span class="s2">[0m"</span>
<span class="p">}</span>
</code></pre></div>
<p>Using these methods I construct one big string object (complete with newline characters) and write it to the screen all at once. Here it is in use to handle alternating colors as well as highlighting a line if the given search string is found and appending to my final output string <code>$ResultString</code>:</p>
<div class="codehilite"><pre><span></span><code><span class="nb">Out-String</span> <span class="n">-Width</span> <span class="nv">$Host</span><span class="p">.</span><span class="n">UI</span><span class="p">.</span><span class="n">RawUI</span><span class="p">.</span><span class="n">WindowSize</span><span class="p">.</span><span class="n">Width</span> <span class="n">-Stream</span> <span class="p">|</span> <span class="k">ForEach</span><span class="n">-Object</span> <span class="p">{</span>
<span class="c"># Handle special coloring here</span>
<span class="nv">$Row</span> <span class="p">+=</span> <span class="n">1</span>
<span class="k">if</span> <span class="p">(</span> <span class="nv">$filter</span> <span class="o">-and</span> <span class="o">-not</span> <span class="nv">$StateData</span><span class="p">.</span><span class="n">SpidFilter</span> <span class="o">-and</span> <span class="nv">$_</span><span class="p">.</span><span class="n">ToLower</span><span class="p">().</span><span class="n">Contains</span><span class="p">(</span><span class="s2">"</span><span class="p">$(</span><span class="nv">$filter</span><span class="p">.</span><span class="n">ToLower</span><span class="p">())</span><span class="s2">"</span><span class="p">)</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$ResultString</span> <span class="p">+=</span> <span class="s2">"</span><span class="p">$(</span><span class="n">color</span> <span class="nv">$_</span> <span class="s2">"black"</span> <span class="s2">"white"</span><span class="p">)</span><span class="se">`n</span><span class="s2">"</span>
<span class="p">}</span> <span class="k">elseif</span> <span class="p">(</span> <span class="nv">$Row</span> <span class="p">%</span> <span class="n">2</span> <span class="o">-eq</span> <span class="n">1</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$ResultString</span> <span class="p">+=</span> <span class="s2">"</span><span class="p">$(</span><span class="n">color</span> <span class="nv">$_</span> <span class="s2">"cyan"</span> <span class="s2">"default"</span><span class="p">)</span><span class="se">`n</span><span class="s2">"</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nv">$ResultString</span> <span class="p">+=</span> <span class="s2">"</span><span class="p">$(</span><span class="nv">$_</span><span class="p">)</span><span class="se">`n</span><span class="s2">"</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>In the example I am using the <code>$Row</code> variable to determine if I am on an even or odd row and coloring accordingly.</p>
<h2>Display fields and sorting</h2>
<p>I went through a few iterations before I landed on an interesting method of setting the field display and sorting options that control each of the views. As I mentioned in the previous section, to draw the display I just output one big string. That string is created by pipeing the results from the data process into <code>Sort-Object</code> and <code>Format-Table</code>. To make it easy to add new views using this model, most of the view and display options are stored in two different hashtables: <code>$DisplayColumns</code> and <code>$SortOptions</code>. Each hashtable contains a number of other hashtables I then use to pass arguments to <code>Sort-object</code> and <code>Format-Table</code> via splatting. Here is a snippet of the <code>$DisplayColumns</code>:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$DisplayColumns</span> <span class="p">=</span> <span class="p">@{</span>
<span class="s1">'Object Tracking'</span> <span class="p">=</span> <span class="p">(</span>
<span class="s1">'object'</span><span class="p">,</span>
<span class="s1">'workers'</span><span class="p">,</span>
<span class="s1">'blocked'</span><span class="p">,</span>
<span class="s1">'cpu'</span><span class="p">,</span>
<span class="p">@{</span>
<span class="n">Name</span> <span class="p">=</span> <span class="s1">'mem_mb'</span>
<span class="n">Expression</span> <span class="p">=</span> <span class="p">{</span> <span class="s2">"{0:0.00}"</span> <span class="o">-f</span> <span class="nv">$_</span><span class="p">.</span><span class="n">mem_mb</span> <span class="p">}</span>
<span class="n">Alignment</span> <span class="p">=</span> <span class="s2">"right"</span>
<span class="p">},</span>
<span class="p">@{</span>
<span class="n">Name</span> <span class="p">=</span> <span class="s1">'tempdb_mb'</span>
<span class="n">Expression</span> <span class="p">=</span> <span class="p">{</span> <span class="s2">"{0:0.00}"</span> <span class="o">-f</span> <span class="nv">$_</span><span class="p">.</span><span class="n">tempdb_mb</span> <span class="p">}</span>
<span class="n">Alignment</span> <span class="p">=</span> <span class="s2">"right"</span>
<span class="p">},</span>
<span class="p">@{</span>
<span class="n">Name</span> <span class="p">=</span> <span class="s1">'lread_mb'</span>
<span class="n">Expression</span> <span class="p">=</span> <span class="p">{</span> <span class="s2">"{0:0.00}"</span> <span class="o">-f</span> <span class="nv">$_</span><span class="p">.</span><span class="n">lread_mb</span> <span class="p">}</span>
<span class="n">Alignment</span> <span class="p">=</span> <span class="s2">"right"</span>
<span class="p">}</span>
<span class="p">)</span>
<span class="s1">'Waits'</span> <span class="p">=</span> <span class="p">(</span>
<span class="s1">'x'</span><span class="p">,</span>
<span class="s1">'spid'</span><span class="p">,</span>
<span class="p">@{</span>
<span class="n">Name</span><span class="p">=</span><span class="s1">'duration'</span>
<span class="n">Expression</span><span class="p">={(</span><span class="no">[timespan]</span><span class="p">::</span><span class="n">fromseconds</span><span class="p">(</span><span class="nv">$_</span><span class="p">.</span><span class="n">dur_sec</span><span class="p">)).</span><span class="n">ToString</span><span class="p">(</span><span class="s1">'d\.hh\:mm\:ss'</span><span class="p">)}</span>
<span class="n">Alignment</span> <span class="p">=</span> <span class="s2">"right"</span>
<span class="p">},</span>
<span class="s1">'block'</span><span class="p">,</span>
<span class="s1">'status'</span><span class="p">,</span>
<span class="s1">'user'</span><span class="p">,</span>
<span class="s1">'database'</span><span class="p">,</span>
<span class="s1">'program_name'</span><span class="p">,</span>
<span class="s1">'command'</span><span class="p">,</span>
<span class="s1">'wt_ms'</span><span class="p">,</span>
<span class="s1">'wt_type'</span><span class="p">,</span>
<span class="s1">'wt_rsrc'</span><span class="p">,</span>
<span class="s1">'open_tran'</span>
<span class="p">)</span>
<span class="p">}</span>
</code></pre></div>
<p>Here you can see we have two entries (there are more in the actual code), <code>Object Tracking</code> and <code>Waits</code>. Based on the view the user chooses I just store the entry from <code>$DisplayColumns</code> in a separate variable and pass it to <code>Format-Table</code>:</p>
<div class="codehilite"><pre><span></span><code><span class="c"># Set the sort and display options</span>
<span class="nv">$SortOpt</span> <span class="p">=</span> <span class="nv">$SortOptions</span><span class="p">[</span><span class="s2">"</span><span class="p">$(</span><span class="nv">$StateData</span><span class="p">.</span><span class="n">DisplayMode</span><span class="p">)</span><span class="s2">"</span><span class="p">]</span>
<span class="nv">$DisplayOpt</span> <span class="p">=</span> <span class="nv">$DisplayColumns</span><span class="p">[</span><span class="s2">"</span><span class="p">$(</span><span class="nv">$StateData</span><span class="p">.</span><span class="n">DisplayMode</span><span class="p">)$(</span><span class="nv">$SubDisplayMode</span><span class="p">)</span><span class="s2">"</span><span class="p">]</span>
<span class="p">...</span>
<span class="nb">Sort-Object</span> <span class="n">-Property</span> <span class="nv">$SortOpt</span> <span class="p">|</span> <span class="nb">Select-Object</span> <span class="n">-First</span> <span class="nv">$max_display</span> <span class="p">|</span> <span class="p">`</span>
<span class="nb">Format-Table</span> <span class="n">-Property</span> <span class="nv">$DisplayOpt</span> <span class="n">-Wrap</span> <span class="p">|</span> <span class="p">`</span>
<span class="nb">Out-String</span> <span class="n">-Width</span> <span class="nv">$Host</span><span class="p">.</span><span class="n">UI</span><span class="p">.</span><span class="n">RawUI</span><span class="p">.</span><span class="n">WindowSize</span><span class="p">.</span><span class="n">Width</span> <span class="n">-Stream</span> <span class="p">|</span> <span class="p">`</span>
<span class="c"># This is where we pipe to the colorizing code from the previous section</span>
</code></pre></div>
<p>While I still have some code to clean up, this method has worked great and makes it really easy to add new views.</p>
<h1>final thoughts</h1>
<p>Like I said earlier, if I could do this project over again (knowing it would turn into a useful tool) I would have likely written in something like Python, but it turned out to be a highly educational experience. The techniques I used while creating this tool have been reused a lot in other code in our various internal modules so beyond the educational value it has also served to improve our code quality (or at least the UX quality) in general. Please check out the code for this project over at <a href="https://github.com/channeladvisor/sqltop">Github</a>. I'd love feedback on things I could do better, or new features folks would like to see.</p>memory-optimized tempdb metadata bug2021-06-30tempdb_bugLet's look at an interesting bug in SQL Server 2019 related to the new Memory-Optimized TempDB Metadata feature!<p>Hey folks! Today I'll share another bug we found recently when testing some new SQL Server 2019 instances in our development environment. This one is a bit concerning as it could affect a lot of shops and the bug presents itself during a pretty common use case.</p>
<h1>The Setup</h1>
<p>All you need to reproduce this bug is:</p>
<ul>
<li>SQL Server 2019 instance with <code>Memory-Optimized TempDB Metadata</code> enabled</li>
<li>database with a memory-optimized filegroup</li>
<li>A memory-optimized table (doesn't have to be durable) with a non-clustered index on it </li>
</ul>
<p>When this issue first started popping up it was on a SQL Server 2019 VM, but I was able to reproduce it using a fresh container using Docker as well, so you should be able to do the same.</p>
<p>To get our setup ready, we need a database to work with:</p>
<div class="codehilite"><pre><span></span><code><span class="k">CREATE</span><span class="w"> </span><span class="k">DATABASE</span><span class="w"> </span><span class="p">[</span><span class="n">testdatabase</span><span class="p">];</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">ALTER</span><span class="w"> </span><span class="k">DATABASE</span><span class="w"> </span><span class="p">[</span><span class="n">testdatabase</span><span class="p">]</span><span class="w"> </span><span class="k">ADD</span><span class="w"> </span><span class="n">FILEGROUP</span><span class="w"> </span><span class="p">[</span><span class="n">memopt</span><span class="p">]</span><span class="w"> </span><span class="k">CONTAINS</span><span class="w"> </span><span class="n">MEMORY_OPTIMIZED_DATA</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">ALTER</span><span class="w"> </span><span class="k">DATABASE</span><span class="w"> </span><span class="p">[</span><span class="n">testdatabase</span><span class="p">]</span><span class="w"> </span><span class="k">ADD</span><span class="w"> </span><span class="n">FILE</span><span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">NAME</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">N</span><span class="s1">'memopt'</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">FILENAME</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">N</span><span class="s1">'/var/opt/mssql/data/memopt'</span><span class="w"></span>
<span class="p">)</span><span class="w"> </span><span class="k">TO</span><span class="w"> </span><span class="n">FILEGROUP</span><span class="w"> </span><span class="p">[</span><span class="n">memopt</span><span class="p">]</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
</code></pre></div>
<blockquote>
<p>My demo code was run on a Linux-based container, hence the odd file paths. If you don't have access to run containers you will need to change the file path above to a valid path on the instance.</p>
</blockquote>
<p>Now let's create a schema-only memory-optimized table:</p>
<div class="codehilite"><pre><span></span><code><span class="c1">-- Create memory-optimized table</span>
<span class="n">USE</span><span class="w"> </span><span class="p">[</span><span class="n">testdatabase</span><span class="p">];</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">dbo</span><span class="p">.[</span><span class="n">testtable</span><span class="p">]</span><span class="w"></span>
<span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="p">[</span><span class="n">id</span><span class="p">]</span><span class="w"> </span><span class="p">[</span><span class="nb">int</span><span class="p">]</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="k">INDEX</span><span class="w"> </span><span class="p">[</span><span class="n">IX_testtable_id</span><span class="p">]</span><span class="w"> </span><span class="n">NONCLUSTERED</span><span class="w"> </span>
<span class="w"> </span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="p">[</span><span class="n">id</span><span class="p">]</span><span class="w"> </span><span class="k">ASC</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="p">)</span><span class="k">WITH</span><span class="w"> </span><span class="p">(</span><span class="w"> </span><span class="n">MEMORY_OPTIMIZED</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="p">,</span><span class="w"> </span><span class="n">DURABILITY</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">SCHEMA_ONLY</span><span class="w"> </span><span class="p">)</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
</code></pre></div>
<p>Last but not least we need to enable the <code>Memory-Optimized TempDB Metadata</code> feature:</p>
<div class="codehilite"><pre><span></span><code><span class="k">EXEC</span><span class="w"> </span><span class="n">sp_configure</span><span class="w"> </span><span class="s1">'Show Advanced Options'</span><span class="p">,</span><span class="mi">1</span><span class="w"></span>
<span class="n">RECONFIGURE</span><span class="w"></span>
<span class="k">EXEC</span><span class="w"> </span><span class="n">sp_configure</span><span class="w"> </span><span class="s1">'tempdb metadata memory-optimized'</span><span class="p">,</span><span class="mi">1</span><span class="w"></span>
<span class="n">RECONFIGURE</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
</code></pre></div>
<p>After this is enabled you need to reboot the system to apply the change.</p>
<h1>The repro</h1>
<p>Ok, we've got everything set up, now let's run a query and see what happens:</p>
<div class="codehilite"><pre><span></span><code><span class="n">USE</span><span class="w"> </span><span class="p">[</span><span class="n">testdatabase</span><span class="p">];</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">SELECT</span><span class="w"> </span><span class="o">*</span><span class="w"></span>
<span class="k">FROM</span><span class="w"> </span><span class="n">sys</span><span class="p">.</span><span class="n">indexes</span><span class="w"> </span><span class="n">indexes</span><span class="w"> </span>
<span class="w"> </span><span class="k">LEFT</span><span class="w"> </span><span class="k">OUTER</span><span class="w"> </span><span class="k">JOIN</span><span class="w"> </span><span class="n">sys</span><span class="p">.</span><span class="n">stats</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">stats</span><span class="w"></span>
<span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="n">indexes</span><span class="p">.[</span><span class="n">object_id</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">stats</span><span class="p">.[</span><span class="n">object_id</span><span class="p">]</span><span class="w"></span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">indexes</span><span class="p">.[</span><span class="n">index_id</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">stats</span><span class="p">.[</span><span class="n">stats_id</span><span class="p">]</span><span class="w"> </span>
<span class="w"> </span><span class="k">LEFT</span><span class="w"> </span><span class="k">OUTER</span><span class="w"> </span><span class="k">JOIN</span><span class="w"> </span><span class="n">sys</span><span class="p">.</span><span class="n">partitions</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">partitions</span><span class="w"></span>
<span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="n">indexes</span><span class="p">.[</span><span class="n">object_id</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">partitions</span><span class="p">.[</span><span class="n">object_id</span><span class="p">]</span><span class="w"></span>
<span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">indexes</span><span class="p">.</span><span class="n">index_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">partitions</span><span class="p">.</span><span class="n">index_id</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
</code></pre></div>
<p>If everything broke like it should have you should now see an error message (and one that might be familiar if you work with Memory-Optimized OLTP features):</p>
<div class="codehilite"><pre><span></span><code>Msg 41317, Level 16, State 0, Line 3
A user transaction that accesses memory-optimized tables or natively compiled modules cannot access more than
one user database or databases model and msdb, and it cannot write to master.
Commands completed successfully.
</code></pre></div>
<p>If you disable <code>Memory-Optimized TempDB Metadata</code>, reboot the server, and re-run the query, you will no longer see this error.</p>
<h1>Conclusion</h1>
<p>We found this issue after enabling this new feature on a test instance. The errors were being generated by Ola's Index Maintenance scripts, so this isn't really an extreme edge-case issue like we typically run into. It was easy to find the statement that generated the error but my co-worker Pablo Lozano (<a href="https://twitter.com/sqlozano">T</a>|<a href="http://www.sqlozano.com/">B</a>) simplified it and found that it specifically occurs when you JOIN to the <code>sys.stats</code> table from another table. This seems like a major issue taking into account the number of people that use Ola's scripts. I currently have a support case in with Microsoft so hopefully this will be resolved soon.</p>baselining sql server with the tig stack2021-07-13stigIn this post I introduce a quick method to spin up a quick TIG (Telegraf, InfluxDB, Grafana) stack to collect baseline performance metrics from SQL Server.<blockquote>
<p>This post is part of T-SQL Tuesday. Thanks to my buddy Anthony Nocentino for hosting this month's T-SQL Tuesday! Anthony is big in the container space so this month's posts are answering the question "What have you been up to with containers?". Check out his post <a href="https://www.centinosystems.com/blog/sql/t-sql-tuesday-140-what-have-you-been-up-to-with-containers/">here</a>. </p>
</blockquote>
<p>In this post I am going to introduce you to the TIG stack (Telegraf, InfluxDB, Grafana) as a way to collect performance baseline metrics from your SQL Server instances. I will also introduce an open source project I created to get you off the ground and running in a few minutes. If you already use this stack, this project can be a great way to test new versions of the various components, and a great place to do local development on new dashboards.</p>
<h1>tig?</h1>
<p>Outside of an <a href="/news-01">older post</a> when I released the re-write of the Telegraf plugin for SQL Server, I haven't said much about the TIG stack. The TIG stack is a collection of 3 open source projects that combine to create a highly scalable performance metric capturing system. </p>
<ul>
<li><a href="https://github.com/influxdata/telegraf">Telegraf</a> is a metrics collector agent that can run either directly on the host it is collecting data from, or from a central location. Telegraf is responsible for gathering metric data and writing it to InfluxDB.</li>
<li><a href="https://github.com/influxdata/influxdb">InfluxDB</a> is a high-performance time-series database designed specifically to ingest metrics and time-based data.</li>
<li><a href="https://github.com/grafana/grafana">Grafana</a> is the final piece of the puzzle providing a data visualization layer for creating complex dashboards. Grafana queries the data in InfluxDB and generates highly configurable charts, graphs, and tables to display the data in customizable dashboards. </li>
</ul>
<p><img alt="Example Grafana Dashboard" src="/img/stig_grafana.png" /></p>
<h1>why?</h1>
<p>Lots of folks wonder <strong>why</strong> I would go through the trouble of building out a system when so many vendors have already solved the problem of collecting baseline metrics. The answer at the time was simple: cost. With my setup I could monitor close to 600 instances (including dev) for $3,000 USD <strong>per year</strong>. That includes data retention of ~2 years! Are there some administration costs as far as my time is concerned? Of course. In the begining things were a little rough as I learned more about InfluxDB, but once things were configured correctly the most work I've had to do is to expand the size of the data drive as we started collecting more metrics.</p>
<blockquote>
<p>If you want to hear more thoughts on monitoring and vendor-based solutions, check out <a href="https://youtu.be/pwGBXR3kAh4">this episode</a> of Mixed Extents!</p>
</blockquote>
<h1>my setup</h1>
<p>As I said above, where I work we gather detailed performance metrics on all of our 600+ instances every 30 seconds. Costs have gone up a little since I initially set everything up to accommodate for more data points, and an expanded retention period of ~5 years. Generally though, we run a Telegraf client on each machine, have a single InfluxDB instance in AWS, and a single Grafana instance. Generally speaking you could easily run Grafana and InfluxDB on the same box. The big limiting factor here is memory, so I have plenty of CPU to spare on the InfluxDB instance. Since Grafana is simply acting as a data proxy between the user and the InfluxDB server, you could get away with running it on a free-tier server for quite a while without much issue. Look for future posts where I share a bit more about the operational day-to-day of maintaining this configuration.</p>
<h1>enter stig</h1>
<p><a href="https://github.com/m82labs/stig">STIG</a> is a docker-compose based method to quickly spin up the infrastructure you need to start playing around with a SQL Server TIG stack. With a simple config tweak you can have a working TIG stack running on a test server (or your laptop) in a few minutes. The first thing you need to do is clone the repo:</p>
<div class="codehilite"><pre><span></span><code>git clone https://github.com/m82labs/stig.git
</code></pre></div>
<p>There are a few directories and files in the project, but really only one you have to worry about to get things up and running:</p>
<ul>
<li><strong>config</strong> - User configurable values are set in the config file <code>userconfig.yaml</code> in this directory</li>
<li><strong>grafana</strong> - The datasources and sample dashboard for Grafana are located here, you should not need to alter these</li>
<li><strong>src</strong> - The source code for a simple python script that uses <code>userconfig.yaml</code> to generate a working <code>telegraf.conf</code> file on build</li>
<li><strong>telegraf</strong> - Contains the <code>telegraf.conf.temp</code> file. This file should typically not be altered unless you really know what you are doing, as it is just a template file used by a python script to create the final config.</li>
</ul>
<h1>configuration</h1>
<p>Currently the only configuration a you have to worry about is the connection details for the SQL Server instance you want to capture data from. To add an instance, open the <code>config/userconfig.yaml</code> file and add any number of hosts in the following format:</p>
<div class="codehilite"><pre><span></span><code><span class="nt">telegraf</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">sql_plugin</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="nt">hosts</span><span class="p">:</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">host</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">192.168.1.2</span><span class="w"></span>
<span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1433</span><span class="w"></span>
<span class="w"> </span><span class="nt">username</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">telegraf</span><span class="w"></span>
<span class="w"> </span><span class="nt">password</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">TelegrafPassword0!</span><span class="w"></span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">host</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">192.168.1.3</span><span class="w"></span>
<span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1433</span><span class="w"></span>
<span class="w"> </span><span class="nt">username</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">telegraf</span><span class="w"></span>
<span class="w"> </span><span class="nt">password</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">TelegrafPassword0!</span><span class="w"></span>
</code></pre></div>
<p>Before you can spin up STIG you need to make sure you also have a <code>telegraf</code> user created on the instances you want to manage. The permissions needed are pretty straightforward:</p>
<div class="codehilite"><pre><span></span><code><span class="n">USE</span><span class="w"> </span><span class="n">master</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">CREATE</span><span class="w"> </span><span class="n">LOGIN</span><span class="w"> </span><span class="p">[</span><span class="n">telegraf</span><span class="p">]</span><span class="w"> </span><span class="k">WITH</span><span class="w"> </span><span class="n">PASSWORD</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">N</span><span class="s1">'<your_strong_password>'</span><span class="p">;</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">GRANT</span><span class="w"> </span><span class="k">VIEW</span><span class="w"> </span><span class="n">SERVER</span><span class="w"> </span><span class="k">STATE</span><span class="w"> </span><span class="k">TO</span><span class="w"> </span><span class="p">[</span><span class="n">telegraf</span><span class="p">];</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
<span class="k">GRANT</span><span class="w"> </span><span class="k">VIEW</span><span class="w"> </span><span class="k">ANY</span><span class="w"> </span><span class="n">DEFINITION</span><span class="w"> </span><span class="k">TO</span><span class="w"> </span><span class="p">[</span><span class="n">telegraf</span><span class="p">];</span><span class="w"></span>
<span class="k">GO</span><span class="w"></span>
</code></pre></div>
<h1>run it!</h1>
<p>Starting up STIG is simple, just change to the root directory of the repo and run: </p>
<div class="codehilite"><pre><span></span><code>docker-compose up
</code></pre></div>
<p>This command will be followed by some output telling you what is happening. This is nice as you can quickly see if you got something wrong, for example:</p>
<div class="codehilite"><pre><span></span><code>Creating volume "stig_influx_data" with default driver
Creating volume "stig_grafana_data" with default driver
Creating stig_influxdb_1 ... done
Creating stig_config_1 ... done
Creating stig_grafana_1 ...
Creating stig_grafana_1 ... error
WARNING: Host is already in use by another container
ERROR: for stig_grafana_1 Cannot start service grafana: driver failed programming external connectivity on endpoint stig_grafana_1 (32b5d854eeb3966fbd42d3a6a5f8076bd36bd0aab4dba0eaddCreating stig_telegraf_1 ... done
ERROR: for grafana Cannot start service grafana: driver failed programming external connectivity on endpoint stig_grafana_1 (32b5d854eeb3966fbd42d3a6a5f8076bd36bd0aab4dba0eadd2eaba2e3919ad9): Error starting userland proxy: listen tcp4 0.0.0.0:8080: bind: address already in use
</code></pre></div>
<p>This just means that something else is already using port 8080, so the Grafana container can't start. There are a number of errors you could run into, but they should all be displayed immediately, and typically be easy to understand.</p>
<p>Assuming everything is up and running, you should now be able to connect to <a href="http://localhost:8080">http://localhost:8080</a> to access your Grafana install. By default the username is <code>stig</code> and the password is <code>stigPass!</code>.</p>
<h2>notes on altering the dashboard</h2>
<p>Since the dashboard that ships with this project is imported on build, you can't modify it directly. Instead, you'll need to make a copy. To make a copy, open the dashboard and click on the settings icon (a gear) at the top:</p>
<p><img alt="Settings gear icon" src="/img/stig_grafana_settings.png" /></p>
<p>You can then choose <code>Save As</code> to make your own copy of the dashboard. If you end up creating a dashboard you think you want to use on another Grafana instance you can export the dashboard to a file by selecting the <code>Share</code> icon at the top and clicking on the <code>Export</code> tab:</p>
<p><img alt="Share menu" src="/img/stig_grafana_share.png" /></p>
<h1>shutting it down</h1>
<p>Stopping STIG is just as easy as starting it:</p>
<div class="codehilite"><pre><span></span><code>docker-compose down
</code></pre></div>
<p>This will gracefully shut all the containers down. The containers are still there though, so issuing another <code>docker-compse up</code> command would get you back to where you were. If you want to completely remove the containers you'll need to issue another command:</p>
<div class="codehilite"><pre><span></span><code>docker-compose rm
</code></pre></div>
<p>This will ask you to confirm and then completely delete the container instances from your machine. This command will NOT remove the data collected by STIG though, I'll talk about that a bit more in the next section.</p>
<h1>notes on persistent data</h1>
<p>This compose file uses named volumes to retain data captured with STIG. You could run an hour worth of capture, destroy the whole setup, and start it up again and still have access to your data. To view the volumes currently set up you can run: </p>
<div class="codehilite"><pre><span></span><code>docker volume ls
</code></pre></div>
<p>This should display a list of volumes, in it you'll see the volumes used by <code>stig</code>:</p>
<div class="codehilite"><pre><span></span><code>DRIVER VOLUME NAME
local stig_grafana_data
local stig_influx_data
</code></pre></div>
<p>You can get more detailed information on these volumes (like the location of the volumes on your local file system) using the <code>docker volume inspect</code> command:</p>
<div class="codehilite"><pre><span></span><code>docker volume inspect stig_influx_data
</code></pre></div>
<p>This will output some useful info:</p>
<div class="codehilite"><pre><span></span><code>[
{
"CreatedAt": "2021-07-07T13:57:50-04:00",
"Driver": "local",
"Labels": {
"com.docker.compose.project": "stig",
"com.docker.compose.version": "1.25.0",
"com.docker.compose.volume": "grafana_data"
},
"Mountpoint": "/var/lib/docker/volumes/stig_grafana_data/_data",
"Name": "stig_grafana_data",
"Options": null,
"Scope": "local"
}
]
</code></pre></div>
<p>This lets us know that the volume itself is storing data locally at <code>/var/lib/docker/volumes/stig_grafana_data/_data</code>, and tells us when it was created. </p>
<p>If you ever want to delete those volumes and reclaim some space, you can easily delete them:</p>
<div class="codehilite"><pre><span></span><code>docker volume rm stig_influx_data stig_grafana_data
</code></pre></div>
<h1>next steps</h1>
<p>While running TIG in production is a bit more involved this project does give you an easy way to get it up and running and start investigating how each component works. Below are some great resources so you can start exploring:</p>
<ul>
<li><a href="https://grafana.com/grafana/resources">Grafana Resources</a> - The Grafana documentation is a great place to start. The docs are well written with a heavy focus on getting things done.</li>
<li><a href="https://docs.influxdata.com/influxdb/v1.8/concepts/">InfluxDB Concepts</a> - This is a specific section of the documentation on important concepts to understand when working with InfluxDB. The rest of the documentation is also fantastic and give you some good foundational knowledge to do more with InfluxDb.</li>
<li><a href="https://docs.influxdata.com/influxdb/v1.8/query_language/continuous_queries/">InfluxDb Continuous Queries</a> - While part of that same documentation I think <code>continuous queries</code> warrant being called out specifically. This is an amazing feature that allows you to easily re-aggregate data in any way you see fit, on a scheduled basis, automatically. One primary use of this feature it to reduce granularity of metrics for long term storage. Say you want to keep the last weeks worth of metrics at a 10 second granularity, the last months at an hourly granularity, and anything older at a 6 hour granularity. With continuous queries you can get all of that with three commands. This allows you to keep long-term trending data while also minimizing storage consumption.</li>
<li><a href="https://github.com/influxdata/telegraf/tree/master/plugins/inputs/sqlserver">Telegraf SQL Server Plugin</a> - A direct link to the SQL Server plugin for Telegraf. This is the plugin that queries your SQL Server instance for metrics and writes them to InfluxDB. If you go one level higher you can see that Telegraf has an enormous number of plugins capable of getting metrics from most popular backend systems.</li>
</ul>
<p>Also, keep your eyes open for more blog posts about using this setup in a large production environment. </p>runspace output, parameters, and variables2021-08-23runspaces-outputHow to handle output, parameters, and variables in runspaces.<p>In a previous post I discussed <a href="/runspaces-explained/">runspaces and instances</a> and explained some of the components involved. In this post, I'll dig a little deeper and show you a simpler way to get data out of a runspace, how to pass parameters in, and how to create shared variables between any number of concurrently executing runspaces.</p>
<blockquote>
<p>In the previous post, I pointed out the differences between runspaces and instances. I have since given up on trying to change how I refer to them, so in this post I will be using the term interchangeably. Also, if you haven't read the previous post, or it's been a while, I would give it a read so you are familiar with the example code.</p>
</blockquote>
<h2>Output</h2>
<p>The typical pattern when developing using runspaces usually involves creating a new PowerShell instance, assigning it a runspace pool, adding a script to execute, and then calling <code>BeginInvoke()</code>. <code>BeginInvoke()</code> then returns an async result object you can store in a variable for later use to retrieve your results by running <code>EndInvoke()</code>. I found this to be a bit confusing, so I set out to find something easier to use. It turns out <code>BeginInvoke()</code> has a number of ways you can call it, one of which allows you to specify an object to store results in.</p>
<p>Here is the example code from the last post illustrating the usual method for getting data out of your runspaces:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$RunspacePool</span> <span class="p">=</span> <span class="no">[RunspaceFactory]</span><span class="p">::</span><span class="n">CreateRunspacePool</span><span class="p">(</span><span class="n">1</span><span class="p">,</span> <span class="n">5</span><span class="p">)</span>
<span class="nv">$RunspacePool</span><span class="p">.</span><span class="n">Open</span><span class="p">()</span>
<span class="nv">$Results</span> <span class="p">=</span> <span class="p">@()</span>
<span class="nv">$Instances</span> <span class="p">=</span> <span class="p">@()</span>
<span class="p">(</span><span class="n">1</span><span class="p">..</span><span class="n">10</span><span class="p">)</span> <span class="p">|</span> <span class="k">ForEach</span><span class="n">-Object</span> <span class="p">{</span>
<span class="nv">$Instance</span> <span class="p">=</span> <span class="no">[powershell]</span><span class="p">::</span><span class="n">Create</span><span class="p">().</span><span class="n">AddScript</span><span class="p">({</span><span class="nb">Get-Random</span><span class="p">})</span>
<span class="nv">$Instance</span><span class="p">.</span><span class="n">RunspacePool</span> <span class="p">=</span> <span class="nv">$RunspacePool</span>
<span class="nv">$Instances</span> <span class="p">+=</span> <span class="nb">New-Object</span> <span class="n">PSObject</span> <span class="n">-Property</span> <span class="p">@{</span>
<span class="n">Instance</span> <span class="p">=</span> <span class="nv">$Instance</span>
<span class="n">State</span> <span class="p">=</span> <span class="nv">$Instance</span><span class="p">.</span><span class="n">BeginInvoke</span><span class="p">()</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nv">$Instances</span> <span class="p">|</span> <span class="k">ForEach</span><span class="n">-Object</span> <span class="p">{</span>
<span class="nv">$Results</span> <span class="p">+=</span> <span class="nv">$_</span><span class="p">.</span><span class="n">Instance</span><span class="p">.</span><span class="n">EndInvoke</span><span class="p">(</span><span class="nv">$_</span><span class="p">.</span><span class="n">State</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div>
<p>In the second to last line, we are running <code>EndInvoke()</code> and storing the output in <code>$Results</code>. But, if you call <code>BeginInvoke()</code> in a slightly different way, you can skip running <code>EndInvoke()</code> altogether:</p>
<div class="codehilite"><pre><span></span><code><span class="n">State</span> <span class="p">=</span> <span class="nv">$Instance</span><span class="p">.</span><span class="n">BeginInvoke</span><span class="p">(</span><span class="nv">$Inputs</span><span class="p">,</span> <span class="nv">$Results</span><span class="p">)</span>
</code></pre></div>
<p>This method takes two objects of type <code>PSDataCollection</code>. The first object <code>$Inputs</code> is a collection of objects that will be passed via the pipeline to the code in your script block, the second <code>$Results</code> captures the pipeline output of your script block.</p>
<blockquote>
<p>I am not going to go into any examples about using the input parameter here, but since it is required I wanted to make sure I explained what it can be used for.</p>
</blockquote>
<p>The cool thing about this method is that the data output by your runspaces is all collected in the <code>$Results</code> object as soon as execution completes. You don't have to call <code>EndInvoke</code> to capture the data later. Here is a full example using this method to replace the method we used in the original script in the <a href="/runspaces-explained/">previous post</a>:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$RunspacePool</span> <span class="p">=</span> <span class="no">[RunspaceFactory]</span><span class="p">::</span><span class="n">CreateRunspacePool</span><span class="p">(</span><span class="n">1</span><span class="p">,</span> <span class="n">5</span><span class="p">)</span>
<span class="nv">$RunspacePool</span><span class="p">.</span><span class="n">Open</span><span class="p">()</span>
<span class="c"># Create pipeline input and output (results) objects</span>
<span class="nv">$Inputs</span> <span class="p">=</span> <span class="nb">New-Object</span> <span class="s1">'System.Management.Automation.PSDataCollection[PSObject]'</span>
<span class="nv">$Results</span> <span class="p">=</span> <span class="nb">New-Object</span> <span class="s1">'System.Management.Automation.PSDataCollection[PSObject]'</span>
<span class="nv">$ScriptBlock</span> <span class="p">=</span> <span class="p">{</span> <span class="nb">Get-Random</span><span class="p">;</span> <span class="p">}</span>
<span class="nv">$Instances</span> <span class="p">=</span> <span class="p">(</span><span class="n">1</span><span class="p">..</span><span class="n">10</span><span class="p">)</span> <span class="p">|</span> <span class="k">ForEach</span><span class="n">-Object</span> <span class="p">{</span>
<span class="nv">$Instance</span> <span class="p">=</span> <span class="no">[powershell]</span><span class="p">::</span><span class="n">Create</span><span class="p">().</span><span class="n">AddScript</span><span class="p">(</span><span class="nv">$ScriptBlock</span><span class="p">)</span>
<span class="nv">$Instance</span><span class="p">.</span><span class="n">RunspacePool</span> <span class="p">=</span> <span class="nv">$RunspacePool</span>
<span class="no">[PSCustomObject]</span><span class="p">@{</span>
<span class="n">Instance</span> <span class="p">=</span> <span class="nv">$Instance</span>
<span class="n">State</span> <span class="p">=</span> <span class="nv">$Instance</span><span class="p">.</span><span class="n">BeginInvoke</span><span class="p">(</span><span class="nv">$Inputs</span><span class="p">,</span><span class="nv">$Results</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">while</span> <span class="p">(</span> <span class="nv">$Instances</span><span class="p">.</span><span class="n">State</span><span class="p">.</span><span class="n">IsCompleted</span> <span class="o">-contains</span> <span class="nv">$False</span><span class="p">)</span> <span class="p">{</span> <span class="nb">Start-Sleep</span> <span class="n">-Milliseconds</span> <span class="n">10</span> <span class="p">}</span>
</code></pre></div>
<p>If you now take a look at <code>$Results</code> it should contain 10 random numbers. I prefer this method as we don't have to loop through and collect the results after everything is done, and it's a little easier to read and understand. </p>
<h2>Parameters</h2>
<p>Running a script via a bunch of runspaces is fantastic if you need to get something done fast, but what if you need to pass some data into the runspace? Like a server address to connect to? </p>
<p>In the code above, the script is being passed into the runspace via <code>$Instance = [powershell]::Create().AddScript($ScriptBlock)</code>. It turns out adding parameters is as easy as adding <code>.AddParameter()</code> to that same line. For this example I'll make the scriptblock being executed a little more complex:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$ScriptBlock</span> <span class="p">=</span> <span class="p">{</span>
<span class="k">param</span><span class="p">(</span>
<span class="no">[int]</span><span class="nv">$Min</span> <span class="p">=</span> <span class="n">0</span><span class="p">,</span>
<span class="no">[int]</span><span class="nv">$Max</span> <span class="p">=</span> <span class="n">100</span>
<span class="p">)</span>
<span class="nb">Get-Random</span> <span class="n">-Maximum</span> <span class="nv">$Max</span> <span class="n">-Minimum</span> <span class="nv">$Min</span>
<span class="p">}</span>
</code></pre></div>
<p>With these changes we can now pass in a minimum and maximum value for our random number. There are two ways we can pass parameter values into the runspace, one is by calling <code>AddParameter()</code> one at a time, the other is by using <a href="/optional_param_splatting/">splatting</a>.</p>
<p>Calling <code>AddParameter()</code> is good for passing a single parameter value into a runspace, but can get a little tedious when you have many parameters. When calling <code>AddParameter()</code> you pass in the name of the parameter followed by the value:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$Instance</span> <span class="p">=</span> <span class="no">[powershell]</span><span class="p">::</span><span class="n">Create</span><span class="p">().</span><span class="n">AddScript</span><span class="p">(</span><span class="nv">$ScriptBlock</span><span class="p">).</span><span class="n">AddParameter</span><span class="p">(</span><span class="s2">"Min"</span><span class="p">,</span><span class="n">100</span><span class="p">).</span><span class="n">AddParameter</span><span class="p">(</span><span class="s2">"Max"</span><span class="p">,</span><span class="n">1000</span><span class="p">)</span>
</code></pre></div>
<p>As you can see, you can string multiple calls to <code>AddParameter()</code> together to pass in multiple parameters. This can be made a little neater using parameter splatting. To use splatting, define a hashtable <code>@{}</code> with entries for each of the parameters you want to pass in, and then make single call to <code>AddParameters()</code>:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$Params</span> <span class="p">=</span> <span class="p">@{</span>
<span class="n">Min</span> <span class="p">=</span> <span class="n">100</span>
<span class="n">Max</span> <span class="p">=</span> <span class="n">1000</span>
<span class="p">}</span>
<span class="nv">$Instance</span> <span class="p">=</span> <span class="no">[powershell]</span><span class="p">::</span><span class="n">Create</span><span class="p">().</span><span class="n">AddScript</span><span class="p">(</span><span class="nv">$ScriptBlock</span><span class="p">).</span><span class="n">AddParameters</span><span class="p">(</span><span class="nv">$Params</span><span class="p">)</span>
</code></pre></div>
<p>The important part here is that you use <code>AddParameters()</code> (plural) instead of <code>AddParameter()</code>.</p>
<h2>Shared Data</h2>
<p>To finish off this post let's take a look at sharing data in runspaces. Running tasks in parallel is great but sometimes those tasks need to work on a shared set of data. Luckily there are a set of <em>synchronized</em> .NET data types designed specifically for thread-safe operations (like those executed in multiple runspaces). Creating a thread-safe variable is simple: </p>
<div class="codehilite"><pre><span></span><code><span class="nv">$SharedData</span> <span class="p">=</span> <span class="no">[System.Collections.ArrayList]</span><span class="p">::</span><span class="n">Synchronized</span><span class="p">(</span><span class="no">[System.Collections.ArrayList]</span><span class="p">::</span><span class="n">new</span><span class="p">())</span>
</code></pre></div>
<p>That's it! We now have a synchronized <code>ArrayList</code> we can operate on in multiple runspaces at a time.</p>
<blockquote>
<p><code>ArrayList</code> is just one of the synchronized types available. To see a full list, check out the MSDN Documentation on <a href="https://docs.microsoft.com/en-us/dotnet/api/system.collections?view=net-5.0">System.Collections</a>. Many of the collections support <code>Synchronized()</code>.</p>
</blockquote>
<p>We have our variable, so what can we do now? One interesting example involves working through a queue of work using a limited runspace pool. For example, queuing up a list of work to do and trying to process that work using no more than 4 runspaces that pull work from the queue as they finish other work. To give the runspace access to the synchronized queue we just pass it in like any other variable using <code>AddParameter()</code>:</p>
<div class="codehilite"><pre><span></span><code><span class="c"># Create an empty synchronized queue</span>
<span class="nv">$ServerQueue</span> <span class="p">=</span> <span class="no">[System.Collections.Queue]</span><span class="p">::</span><span class="n">Synchronized</span><span class="p">(</span><span class="no">[System.Collections.Queue]</span><span class="p">::</span><span class="n">new</span><span class="p">())</span>
<span class="c"># Add some fake servers to the queue</span>
<span class="n">1</span><span class="p">..</span><span class="n">25</span> <span class="p">|</span> <span class="k">ForEach</span><span class="n">-Object</span> <span class="p">{</span>
<span class="nv">$ServerQueue</span><span class="p">.</span><span class="n">Enqueue</span><span class="p">(</span><span class="s2">"Server</span><span class="p">$(</span><span class="nv">$_</span><span class="p">)</span><span class="s2">"</span><span class="p">)</span>
<span class="p">}</span>
<span class="nv">$QueueCount</span> <span class="p">=</span> <span class="nv">$ServerQueue</span><span class="p">.</span><span class="n">Count</span>
<span class="c"># Create some fake work</span>
<span class="nv">$ScriptBlock</span> <span class="p">=</span> <span class="p">{</span>
<span class="k">param</span> <span class="p">(</span>
<span class="nv">$ServerQueue</span>
<span class="p">)</span>
<span class="k">while</span><span class="p">(</span> <span class="nv">$ServerQueue</span><span class="p">.</span><span class="n">Count</span> <span class="o">-gt</span> <span class="n">0</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$Server</span> <span class="p">=</span> <span class="nv">$ServerQueue</span><span class="p">.</span><span class="n">Dequeue</span><span class="p">()</span>
<span class="nb">Write-Output</span> <span class="s2">"Starting work on </span><span class="p">$(</span><span class="nv">$Server</span><span class="p">)</span><span class="s2">"</span>
<span class="nb">Start-Sleep</span> <span class="n">-Seconds</span> <span class="p">$(</span><span class="nb">Get-Random</span> <span class="n">-Minimum</span> <span class="n">1</span> <span class="n">-Maximum</span> <span class="n">4</span><span class="p">)</span>
<span class="nb">Write-Output</span> <span class="s2">"Work Complete"</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nv">$Inputs</span> <span class="p">=</span> <span class="nb">New-Object</span> <span class="s1">'System.Management.Automation.PSDataCollection[PSObject]'</span>
<span class="nv">$Results</span> <span class="p">=</span> <span class="nb">New-Object</span> <span class="s1">'System.Management.Automation.PSDataCollection[PSObject]'</span>
<span class="c">#Spin up 4 runspaces to process the work</span>
<span class="nv">$Instances</span> <span class="p">=</span> <span class="p">@()</span>
<span class="p">(</span><span class="n">1</span><span class="p">..</span><span class="n">4</span><span class="p">)</span> <span class="p">|</span> <span class="k">ForEach</span><span class="n">-Object</span> <span class="p">{</span>
<span class="nv">$Instance</span> <span class="p">=</span> <span class="no">[powershell]</span><span class="p">::</span><span class="n">Create</span><span class="p">().</span><span class="n">AddScript</span><span class="p">(</span><span class="nv">$ScriptBlock</span><span class="p">).</span><span class="n">AddParameter</span><span class="p">(</span><span class="s1">'ServerQueue'</span><span class="p">,</span> <span class="nv">$ServerQueue</span><span class="p">)</span>
<span class="nv">$Instances</span> <span class="p">+=</span> <span class="nb">New-Object</span> <span class="n">PSObject</span> <span class="n">-Property</span> <span class="p">@{</span>
<span class="n">Instance</span> <span class="p">=</span> <span class="nv">$Instance</span>
<span class="n">State</span> <span class="p">=</span> <span class="nv">$Instance</span><span class="p">.</span><span class="n">BeginInvoke</span><span class="p">(</span><span class="nv">$Inputs</span><span class="p">,</span><span class="nv">$Results</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c"># Lets loop and wait for work to complete</span>
<span class="k">while</span> <span class="p">(</span> <span class="nv">$Instances</span><span class="p">.</span><span class="n">State</span><span class="p">.</span><span class="n">IsCompleted</span> <span class="o">-contains</span> <span class="nv">$False</span><span class="p">)</span> <span class="p">{</span>
<span class="c"># Report the servers left in the queue</span>
<span class="nb">Write-Host</span> <span class="s2">"Server(s) Remaining: </span><span class="p">$(</span><span class="nv">$ServerQueue</span><span class="p">.</span><span class="n">Count</span><span class="p">)</span><span class="s2">/</span><span class="p">$(</span><span class="nv">$QueueCount</span><span class="p">)</span><span class="s2">"</span>
<span class="nb">Start-Sleep</span> <span class="n">-Milliseconds</span> <span class="n">1000</span>
<span class="p">}</span>
</code></pre></div>
<p>In this example, we first create an empty queue and then fill the using some fake server names, and then keep track of the original queue count for display purposes. The scriptblock in this example is a little interesting:</p>
<div class="codehilite"><pre><span></span><code><span class="k">param</span> <span class="p">(</span>
<span class="nv">$ServerQueue</span>
<span class="p">)</span>
<span class="k">while</span><span class="p">(</span> <span class="nv">$ServerQueue</span><span class="p">.</span><span class="n">Count</span> <span class="o">-gt</span> <span class="n">0</span> <span class="p">)</span> <span class="p">{</span>
<span class="nv">$Server</span> <span class="p">=</span> <span class="nv">$ServerQueue</span><span class="p">.</span><span class="n">Dequeue</span><span class="p">()</span>
<span class="nb">Write-Output</span> <span class="s2">"Starting work on </span><span class="p">$(</span><span class="nv">$Server</span><span class="p">)</span><span class="s2">"</span>
<span class="nb">Start-Sleep</span> <span class="n">-Seconds</span> <span class="p">$(</span><span class="nb">Get-Random</span> <span class="n">-Minimum</span> <span class="n">1</span> <span class="n">-Maximum</span> <span class="n">4</span><span class="p">)</span>
<span class="nb">Write-Output</span> <span class="s2">"Work Complete"</span>
<span class="p">}</span>
</code></pre></div>
<p>In this script we are sitting in a while loop and looping while the queue has data in it:</p>
<div class="codehilite"><pre><span></span><code><span class="k">while</span><span class="p">(</span> <span class="nv">$ServerQueue</span><span class="p">.</span><span class="n">Count</span> <span class="o">-gt</span> <span class="n">0</span> <span class="p">)</span> <span class="p">{</span>
</code></pre></div>
<p>Within the loop we <code>Dequeue()</code> the next item on the queue and do a little fake work:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">$Server</span> <span class="p">=</span> <span class="nv">$ServerQueue</span><span class="p">.</span><span class="n">Dequeue</span><span class="p">()</span>
<span class="nb">Write-Output</span> <span class="s2">"Starting work on </span><span class="p">$(</span><span class="nv">$Server</span><span class="p">)</span><span class="s2">"</span>
<span class="nb">Start-Sleep</span> <span class="n">-Seconds</span> <span class="p">$(</span><span class="nb">Get-Random</span> <span class="n">-Minimum</span> <span class="n">1</span> <span class="n">-Maximum</span> <span class="n">4</span><span class="p">)</span>
</code></pre></div>
<p>This means that this "worker" script will run until the queue is empty. </p>
<p>The rest of this script looks as you would expect, we create some instances and start some work. The big difference here is that we are only creating 4 instances to run our work on:</p>
<div class="codehilite"><pre><span></span><code><span class="p">(</span><span class="n">1</span><span class="p">..</span><span class="n">4</span><span class="p">)</span> <span class="p">|</span> <span class="k">ForEach</span><span class="n">-Object</span> <span class="p">{</span>
</code></pre></div>
<p>This approach can be useful if you have a lot of work to get through and don't want to use all the memory necessary to spin up 100+ instances when you are only going to allow a few to run at a time. So this queue method allows us to limit our runing instances while still allowing us to churn through all the work that needs to be done. </p>
<h1>Conclusion</h1>
<p>In this post, we covered some of the ways to move data around when using runspaces. We also discussed the <em>synchronized</em> data types and how they can be useful in cases where you require concurrent access to a set of data. All of these tools combined can allow you to create some pretty interesting workflow patterns. In a future post, I'll share a pattern I use that allows me to process work but exit early when errors are detected. </p>
}