Interop 2015: Hardly Enough Theory, Barley Enough Code

Interop 2015: Hardly Enough Theory, Barley Enough Code

• 1.
  intermediate Python techniques forthe automation journey April 2015 Jeremy Schulman @nwkautomaniac Hardly Enough Theory Barely Enough Code
• 2.
  Agenda ● Automation -Why Me? ● Learn key concepts ● Learn cool Pythonic techniques
• 3.
  Why Automate? Don’t bethe bottleneck Empower your team Increase agility and productivity Keep the business running Automate the mundane Free up your time Improve uptime and reliability Your time and your expertise
• 4.
  Theory ● Composers, constructors andconsumers ● Decomposition ● What matters most ● Zero, one, or infinity What You will Learn Code ● Iterators and generators ● Files - JSON, YAML, CSV ● Validation - JSON schema ● Templating - Jinja2 ○ Basics ○ Programming logic ○ Custom extensions
• 5.
  Theory Just a fewslides ...
• 6.
  Role Relationships Design thingsproviding instructions to others Combination of art and science Creative processes and best-practices Composers Build finished goods based on the instructions Practiced skills - years to master the craft Specialized domain knowledge and tools Constructors Use finished goods Generally does not require much skill Sometimes requires use of tools / products Consumers
• 7.
  Decomposition Everyone has favorites: ●Modeling techniques ... ● Diagramming techniques ... ● Process techniques ... Breaking a complex problem or system into parts Easier to conceive, understand, program and maintain Also known as factoring Entity Relationship DiagramFlowchart Diagram
• 8.
  Apply Decomposition Consider aswitch configuration file ... ● Create shared and unique data-sets ● Define variables into explicit and derived / calculated items ● Understand cohesion of data and code ● Reuse the data for other purposes, like validation and operations ● Data can be used for multi-vendor Step 1: separate "data" from "code" Step 2: decompose "data" Step 3: decompose "code" Step 4: goto step 2 until manageable Why do this?
• 9.
  What Matters Most? Doesit do what it is supposed to do? Are there even any requirements? Is high availability / fault tolerant a consideration? Functionality How fast does it run? Is speed really even an issue?Performance Does it need to handle the growth of your environment? Do you need to deal with concurrent processing?Scale When it breaks, can you fix it (quickly)? Can other people work on it? What happens if you leave? Maintainability
• 10.
  Zero, One, orInfinity No need to do something so just don't do itZero I must think about everyone else that might use this or have to deal with it one day. Empathy. Infinity Doing it only for myself and I don't need to worry about anyone else using it or what happens when I'm gone One
• 11.
  Code Iterators and Generators
• 12.
  Iterators Loop over things ●list, dictionary, set, tuple ● files - each line in text file ● strings - each character in string ● class objects supporting the iter protocol ● generators - see next slide List iteration code example Example output
• 13.
  Why use Generators? LazyEvaluation Evaluation strategy which delays the evaluation of an expression until its value is needed Benefits ● Reduce memory ● Generate data on-demand ● Generate “infinite” data ● Performance increases ● Use as abstract objects ○ as function args ○ treat like any other iterablehttp://en.wikipedia.org/wiki/Lazy_evaluation
• 14.
  Generators Produce sequence of things ●behave like iterators, but lazy ● generator functions use yield ● generator expressions are comprehensions enclosed by parentheses Generator expression example Generator function example
• 15.
  Code Loading Data: JSON,YAML and CSV
• 16.
  Loading Data fromFiles - JSON Example file Example code Example output
• 17.
  Loading Data fromFiles - YAML Example file Example code Example output
• 18.
  Loading Data fromFiles - CSV Example file Example code Example outputDo you see the differences? 1. id is a string, not integer 2. name is not the key How to fix this?
• 19.
  Loading Data fromFiles - CSV (fixed!) Example code Example output
• 20.
  Demo: Putting itall together Task Load all of the files, of different types, into a single dictionary variable Pythonic techniques ● iterators and generators ● lambda functions ● reduce function ● iterables.chain ● functools.partial
• 21.
  Example Code
• 22.
  Demo: Filtering whileloading Task While reading the data files, filter out records that match a given criteria id >= 20 Pythonic techniques ● generator ● compression ● … extract first/only dict item ● … with if statement
• 23.
  Example Code Example output
• 24.
  Code Templating with Jinja2
• 25.
  Templating with Jinja2- Intro Loaded from text files Created dynamically by program Commonly used for configuration {% if age > 12 %} ... {% endif %} Hello {{ name | capitalize }} Hello {{ name }} Template content is text Programmatic controls Filters and extensions Variable string interpolation
• 26.
  Very Simple Example- Jinja2
• 27.
  Programmatic Controls -Jinja2 {% if age > 20 %} {% else %} {% endif %} if / then / else {% for this in some_iterable %} {% endfor %}loop over iterators {% set myvar = some_func() %}create variables {% include "this file.j2" %} {% include file_name %}include other files
• 28.
  Load template searchingdirectories Jinja2 "Loader" determines how to find the template files Jinja2 "Environment" allows you to control aspects template loading and rendering
• 29.
  Example: Program Control
• 30.
  Custom Filters -Jinja2 Create a sorted key list by using the inner data dictionary Bind the function to the Jinja2 environment "filters" dictionary
• 31.
  Example Output
• 32.
  Including other files- Jinja2
• 33.
  Example Output The state_listvariable using in the template was created in the python code using the groupby function, see next slide ...
• 34.
  Example Code: Report https://docs.python.org/2/library/itertools.html#itertools.groupby
• 35.
  Code Data Validation
• 36.
  Data Validation -JSON Schema Used to validate data Verify the syntactic contents Both inputs and outputs Schema is written in JSON Other schema mechanism: ● ASN.1 - SNMP ● YANG - NETCONF ● XSD - XML Example Data Example Schema
• 37.
  More interesting example DataSchema ● One "record" per file ● Record key is max 16 alpha-chars ● Only the properties: ○ state ○ id ○ job
• 38.
  JSON Schema -Docs http://json-schema.org/latest/json-schema-validation.html#toc
• 39.
  Automation Summary Programming isa tool not a career change It's not about you it's about everyone else Try being right now rather than being right
• 40.
  More? ● JSON -https://docs.python.org/2/library/json.html ● JSON Schema - http://json-schema.org ● YAML - http://pyyaml.org/wiki/PyYAMLDocumentation ● CSV - https://docs.python.org/2/library/csv.html ● Jinja2 - http://jinja.pocoo.org/ ● itertools - https://docs.python.org/2/library/itertools.html ● jq - https://github.com/stedolan/jq ● nwkautomaniac - https://github.com/jeremyschulman/nwkautomaniac
• 41.
  Jeremy Schulman @nwkautomaniac Thank you!