diff --git a/CHANGELOG.md b/CHANGELOG.md index 626275c..8ee18c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,8 @@ Squib follows [semantic versioning](http://semver.org). Features: * `save_pdf` now supports crop marks! These are lines drawn in the margins of a PDF file to help you cut. These can be enabled by setting `crop_marks: true` in your `save_pdf` call. Can be further customized with `crop_margin_bottom`, `crop_margin_left`, `crop_margin_right`, `crop_margin_top`, `crop_marks`, `crop_stroke_color`, `crop_stroke_dash`, and `crop_stroke_width` (#123) * `Squib.configure` allows you to set options programmatically, overriding your config.yml. This is useful for Rakefiles, and will be documented in my upcoming tutorial on workflows. -* `Squib.enable_build_globally` and `Squib.disable_build_globally` are new convenience methods for working with the `SQUIB_BUILD` environment variable. Handy for Rakefiles and Guard sessions for turning certain builds on an off. Also will be in upcoming workflow tutorial. +* `Squib.enable_build_globally` and `Squib.disable_build_globally` are new convenience methods for working with the `SQUIB_BUILD` environment variable. Handy for Rakefiles and Guard sessions for turning certain builds on an off. Also will be documented in upcoming workflow tutorial. +* The import methods `csv` and `xlsx` now return `Squib::DataFrame`, which behaves exactly as before - but has more cool features like being able to do `data.name` instead of `data['name']`. Also: check out `data.to_pretty_text`. Check out the docs. (#156) Bugs: * `showcase` works as expected when using `backend: svg` (#179) diff --git a/docs/build_groups.rst b/docs/build_groups.rst index 3251502..1f7c10b 100644 --- a/docs/build_groups.rst +++ b/docs/build_groups.rst @@ -34,7 +34,9 @@ One adaptation of this is to do the environment setting in a ``Rakefile``. `Rake :language: ruby :linenos: -Thus, you can just run this code on the command line like these:: +Thus, you can just run this code on the command line like these: + +.. code-block:: none $ rake $ rake pnp diff --git a/docs/data.rst b/docs/data.rst index 901458c..d64b0c2 100644 --- a/docs/data.rst +++ b/docs/data.rst @@ -3,10 +3,12 @@ Be Data-Driven with XLSX and CSV Squib supports importing data from ExcelX (.xlsx) files and Comma-Separated Values (.csv) files. Because :doc:`/arrays`, these methods are column-based, which means that they assume you have a header row in your table, and that header row will define the name of the column. -Hash of Arrays --------------- +Squib::DataFrame, or a Hash of Arrays +------------------------------------- -In both DSL methods, Squib will return a ``Hash`` of ``Arrays`` correspoding to each row. Thus, be sure to structure your data like this: +In both DSL methods, Squib will return a "data frame" (literally of type ``Squib::DataFrame``). The best way to think of this is a ``Hash`` of ``Arrays``, where each column is a key in the hash, and every element of each Array represents a data point on a card. + +The data import methods expect you to structure your Excel sheet or CSV like this: * First row should be a header - preferably with concise naming since you'll reference it in Ruby code * Rows should represent cards in the deck @@ -14,7 +16,9 @@ In both DSL methods, Squib will return a ``Hash`` of ``Arrays`` correspoding to Of course, you can always import your game data other ways using just Ruby (e.g. from a REST API, a JSON file, or your own custom format). There's nothing special about Squib's methods in how they relate to ``Squib::Deck`` other than their convenience. -See :doc:`/dsl/xlsx` and :doc:`/dsl/csv` for more details and examples. +See :doc:`/dsl/xlsx` and :doc:`/dsl/csv` for more details and examples on how the data can be imported. + +The ``Squib::DataFrame`` class provides much more than what a ``Hash`` provides, however. The :doc:`/dsl/data_frame` Quantity Explosion ------------------ diff --git a/docs/dsl/data_frame.rst b/docs/dsl/data_frame.rst new file mode 100644 index 0000000..078293e --- /dev/null +++ b/docs/dsl/data_frame.rst @@ -0,0 +1,85 @@ +Squib::DataFrame +================ + +As described in :doc:`/data`, the ``Squib::DataFrame`` is what is returned by Squib's data import methods (:doc:`/dsl/csv` and :doc:`/dsl/xlsx`). + +It behaves like a ``Hash`` of ``Arrays``, so acessing an individual column can be done via the square brackets, e.g. ``data['title']``. + +Here are some other convenience methods in ``Squib::DataFrame`` + +columns become methods +---------------------- + +Through magic of Ruby metaprogramming, every column also becomes a method on the data frame. So these two are equivalent: + +.. code-block:: irb + + irb(main):002:0> data = Squib.csv file: 'basic.csv' + => #[1, 3], "h2"=>[2, 4]}> + irb(main):003:0> data.h1 + => [1, 3] + irb(main):004:0> data['h1'] + => [1, 3] + +#columns +-------- + +Returns an array of the column names in the data frame + +#ncolumns +--------- + +Returns the number of columns in the data frame + +#col?(name) +----------- + +Returns ``true`` if there is column ``name``. + +#row(i) +------- + +Returns a hash of values across all columns in the ``i``th row of the dataframe. Represents a single card. + +#nrows +------ + +Returns the number of rows the data frame has, computed by the maximum length of any column array. + +#to_json +-------- + +Returns a ``json`` representation of the entire data frame. + +#to_pretty_json +--------------- + +Returns a ``json`` representation of the entire data frame, formatted with indentation for human viewing. + +#to_pretty_text +--------------- + +Returns a textual representation of the dataframe that emulates what the information looks like on an individual card. Here's an example: + +.. code-block:: text + + ╭------------------------------------╮ + Name | Mage | + Cost | 1 | + Description | You may cast 1 spell per turn | + Snark | Magic, dude. | + ╰------------------------------------╯ + ╭------------------------------------╮ + Name | Rogue | + Cost | 2 | + Description | You always take the first turn. | + Snark | I like to be sneaky | + ╰------------------------------------╯ + ╭------------------------------------╮ + Name | Warrior | + Cost | 3 | + Description | + Snark | I have a long story to tell to tes | + | t the word-wrapping ability of pre | + | tty text formatting. | + ╰------------------------------------╯ diff --git a/docs/install.rst b/docs/install.rst index f6cc159..98b3eaf 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -13,7 +13,9 @@ Squib works with both x86 and x86_64 versions of Ruby. Typical Install --------------- -Regardless of your OS, installation is:: +Regardless of your OS, installation is + +.. code-block:: none $ gem install squib diff --git a/lib/squib/api/data.rb b/lib/squib/api/data.rb index fe00ff3..c26b744 100644 --- a/lib/squib/api/data.rb +++ b/lib/squib/api/data.rb @@ -3,6 +3,7 @@ require 'csv' require_relative '../args/input_file' require_relative '../args/import' require_relative '../args/csv_opts' +require_relative '../import/data_frame' module Squib @@ -12,7 +13,7 @@ module Squib import = Args::Import.new.load!(opts) s = Roo::Excelx.new(input.file[0]) s.default_sheet = s.sheets[input.sheet[0]] - data = {} + data = Squib::DataFrame.new s.first_column.upto(s.last_column) do |col| header = s.cell(s.first_row, col).to_s header.strip! if import.strip? @@ -39,14 +40,14 @@ module Squib csv_opts = Args::CSV_Opts.new(opts) table = CSV.parse(data, csv_opts.to_hash) check_duplicate_csv_headers(table) - hash = Hash.new + hash = Squib::DataFrame.new table.headers.each do |header| new_header = header.to_s new_header = new_header.strip if import.strip? hash[new_header] ||= table[header] end if import.strip? - new_hash = Hash.new + new_hash = Squib::DataFrame.new hash.each do |header, col| new_hash[header] = col.map do |str| str = str.strip if str.respond_to?(:strip) @@ -78,9 +79,9 @@ module Squib # @api private def explode_quantities(data, qty) - return data unless data.key? qty.to_s.strip + return data unless data.col? qty.to_s.strip qtys = data[qty] - new_data = {} + new_data = Squib::DataFrame.new data.each do |col, arr| new_data[col] = [] qtys.each_with_index do |qty, index| diff --git a/lib/squib/import/data_frame.rb b/lib/squib/import/data_frame.rb new file mode 100644 index 0000000..cb34048 --- /dev/null +++ b/lib/squib/import/data_frame.rb @@ -0,0 +1,108 @@ +# encoding: UTF-8 + +require 'json' +require 'forwardable' + +module Squib + class DataFrame + include Enumerable + + def initialize(hash = {}, def_columns = true) + @hash = hash + columns.each { |col| def_column(col) } if def_columns + end + + def each(&block) + @hash.each(&block) + end + + def [](i) + @hash[i] + end + + def []=(col, v) + @hash[col] = v + def_column(col) + return v + end + + def columns + @hash.keys + end + + def ncolumns + @hash.keys.size + end + + def col?(col) + @hash.key? col + end + + def row(i) + @hash.inject(Hash.new) { |ret, (name, arr)| ret[name] = arr[i]; ret } + end + + def nrows + @hash.inject(0) { |max, (_n, col)| col.size > max ? col.size : max } + end + + def to_json + @hash.to_json + end + + def to_pretty_json + JSON.pretty_generate(@hash) + end + + def to_h + @hash + end + + def to_pretty_text + max_col = columns.inject(0) { |max, c | c.length > max ? c.length : max } + top = " ╭#{'-' * 36}╮\n" + bottom = " ╰#{'-' * 36}╯\n" + str = '' + 0.upto(nrows - 1) do | i | + str += (' ' * max_col) + top + row(i).each do |col, data| + str += "#{col.rjust(max_col)} #{wrap_n_pad(data, max_col)}" + end + str += (' ' * max_col) + bottom + end + return str + end + + private + + def snake_case(str) + str.to_s. + strip. + gsub(/\s+/,'_'). + gsub(/([A-Z]+)([A-Z][a-z])/,'\1_\2'). + gsub(/([a-z]+)([A-Z])/,'\1_\2'). + downcase. + to_sym + end + + def wrap_n_pad(str, max_col) + str.to_s. + concat(' '). # handle nil & empty strings + scan(/.{1,34}/). + map { |s| (' ' * max_col) + " | " + s.ljust(34) }. + join(" |\n"). + lstrip. # initially no whitespace next to key + concat(" |\n") + end + + def def_column(col) + raise "Column #{col} - does not exist" unless @hash.key? col + method_name = snake_case(col) + return if self.class.method_defined?(method_name) #warn people? or skip? + define_singleton_method method_name do + @hash[col] + end + end + + end +end diff --git a/spec/api/api_data_spec.rb b/spec/api/api_data_spec.rb index 15ab0d9..4993eb9 100644 --- a/spec/api/api_data_spec.rb +++ b/spec/api/api_data_spec.rb @@ -3,7 +3,7 @@ require 'spec_helper' describe Squib::Deck do context '#csv' do it 'loads basic csv data' do - expect(Squib.csv(file: csv_file('basic.csv'))).to eq({ + expect(Squib.csv(file: csv_file('basic.csv')).to_h.to_h).to eq({ 'h1' => [1, 3], 'h2' => [2, 4] }) @@ -12,7 +12,7 @@ describe Squib::Deck do it 'collapses duplicate columns and warns' do expect(Squib.logger).to receive(:warn) .with('CSV duplicated the following column keys: h1,h1') - expect(Squib.csv(file: csv_file('dup_cols.csv'))).to eq({ + expect(Squib.csv(file: csv_file('dup_cols.csv')).to_h.to_h).to eq({ 'h1' => [1, 3], 'h2' => [5, 7], 'H2' => [6, 8], @@ -21,7 +21,7 @@ describe Squib::Deck do end it 'strips spaces by default' do - expect(Squib.csv(file: csv_file('with_spaces.csv'))).to eq({ + expect(Squib.csv(file: csv_file('with_spaces.csv')).to_h).to eq({ 'With Spaces' => ['a b c', 3], 'h2' => [2, 4], 'h3' => [3, nil] @@ -29,7 +29,7 @@ describe Squib::Deck do end it 'skips space stripping if told to' do - expect(Squib.csv(strip: false, file: csv_file('with_spaces.csv'))).to eq({ + expect(Squib.csv(strip: false, file: csv_file('with_spaces.csv')).to_h).to eq({ ' With Spaces ' => ['a b c ', 3], 'h2' => [2, 4], 'h3' => [3, nil] @@ -37,14 +37,14 @@ describe Squib::Deck do end it 'explodes quantities' do - expect(Squib.csv(file: csv_file('qty.csv'))).to eq({ + expect(Squib.csv(file: csv_file('qty.csv')).to_h).to eq({ 'Name' => %w(Ha Ha Ha Ho), 'Qty' => [3, 3, 3, 1], }) end it 'explodes quantities on specified header' do - expect(Squib.csv(explode: 'Quantity', file: csv_file('qty_named.csv'))).to eq({ + expect(Squib.csv(explode: 'Quantity', file: csv_file('qty_named.csv')).to_h).to eq({ 'Name' => %w(Ha Ha Ha Ho), 'Quantity' => [3, 3, 3, 1], }) @@ -52,7 +52,7 @@ describe Squib::Deck do it 'loads inline data' do hash = Squib.csv(data: "h1,h2\n1,2\n3,4") - expect(hash).to eq({ + expect(hash.to_h).to eq({ 'h1' => [1, 3], 'h2' => [2, 4] }) @@ -60,7 +60,7 @@ describe Squib::Deck do it 'loads csv with newlines' do hash = Squib.csv(file: csv_file('newline.csv')) - expect(hash).to eq({ + expect(hash.to_h).to eq({ 'title' => ['Foo'], 'level' => [1], 'notes' => ["a\nb"] @@ -70,7 +70,7 @@ describe Squib::Deck do it 'loads custom CSV options' do hash = Squib.csv(file: csv_file('custom_opts.csv'), col_sep: '-', quote_char: '|') - expect(hash).to eq({ + expect(hash.to_h).to eq({ 'x' => ['p'], 'y' => ['q-r'] }) @@ -85,7 +85,7 @@ describe Squib::Deck do 'ha' end end - expect(data).to eq({ + expect(data.to_h).to eq({ 'h1' => [2, 6], 'h2' => %w(ha ha), }) @@ -99,7 +99,7 @@ describe Squib::Deck do value end end - expect(data).to eq({ + expect(data.to_h).to eq({ 'a' => ["foo\nbar", 1], 'b' => [1, "blah\n"], }) @@ -109,7 +109,7 @@ describe Squib::Deck do context '#xlsx' do it 'loads basic xlsx data' do - expect(Squib.xlsx(file: xlsx_file('basic.xlsx'))).to eq({ + expect(Squib.xlsx(file: xlsx_file('basic.xlsx')).to_h).to eq({ 'Name' => %w(Larry Curly Mo), 'General Number' => %w(1 2 3), # general types always get loaded as strings with no conversion 'Actual Number' => [4.0, 5.0, 6.0], # numbers get auto-converted to integers @@ -117,7 +117,7 @@ describe Squib::Deck do end it 'loads xlsx with formulas' do - expect(Squib.xlsx(file: xlsx_file('formulas.xlsx'))).to eq({ + expect(Squib.xlsx(file: xlsx_file('formulas.xlsx')).to_h).to eq({ 'A' => %w(1 2), 'B' => %w(3 4), 'Sum' => %w(4 6), @@ -125,20 +125,20 @@ describe Squib::Deck do end it 'loads xlsm files with macros' do - expect(Squib.xlsx(file: xlsx_file('with_macros.xlsm'))).to eq({ + expect(Squib.xlsx(file: xlsx_file('with_macros.xlsm')).to_h).to eq({ 'foo' => %w(8 10), 'bar' => %w(9 11), }) end it 'strips whitespace by default' do - expect(Squib.xlsx(file: xlsx_file('whitespace.xlsx'))).to eq({ + expect(Squib.xlsx(file: xlsx_file('whitespace.xlsx')).to_h).to eq({ 'With Whitespace' => ['foo', 'bar', 'baz'], }) end it 'does not strip whitespace when specified' do - expect(Squib.xlsx(file: xlsx_file('whitespace.xlsx'), strip: false)).to eq({ + expect(Squib.xlsx(file: xlsx_file('whitespace.xlsx'), strip: false).to_h).to eq({ ' With Whitespace ' => ['foo ', ' bar', ' baz '], }) end @@ -154,7 +154,7 @@ describe Squib::Deck do 'ha' end end - expect(data).to eq({ + expect(data.to_h).to eq({ 'Name' => %w(he he he), 'General Number' => %w(ha ha ha), 'Actual Number' => [8.0, 10.0, 12.0], @@ -162,7 +162,7 @@ describe Squib::Deck do end it 'explodes quantities' do - expect(Squib.xlsx(explode: 'Qty', file: xlsx_file('explode_quantities.xlsx'))).to eq({ + expect(Squib.xlsx(explode: 'Qty', file: xlsx_file('explode_quantities.xlsx')).to_h).to eq({ 'Name' => ['Zergling', 'Zergling', 'Zergling', 'High Templar'], 'Qty' => %w(3 3 3 1), }) diff --git a/spec/import/data_frame_spec.rb b/spec/import/data_frame_spec.rb new file mode 100644 index 0000000..6f7bec5 --- /dev/null +++ b/spec/import/data_frame_spec.rb @@ -0,0 +1,251 @@ +# encoding: UTF-8 + +require 'spec_helper' +require 'squib/import/data_frame' + +describe Squib::DataFrame do + let(:basic) do + { + 'Name' => ['Mage', 'Rogue', 'Warrior'], + 'Cost' => [1, 2, 3], + } + end + + let(:uneven) do + { + 'Name' => ['Mage', 'Rogue', 'Warrior'], + 'Cost' => [1, 2], + } + end + + let(:whitespace) do + { + 'Name \n\r\t' => [' Mage \t\r\n'], + } + end + + let(:defined) do + { + 'nrows' => [1,2,3], + } + end + + context 'is Enumerable and like a hash, so it' do + it 'responds to each' do + expect(subject).to respond_to(:each) + end + + it 'responds to any?' do + expect(subject.any?).to be false + end + + it 'responds to []' do + data = Squib::DataFrame.new basic + expect(data['Cost']).to eq([1, 2, 3]) + end + + it 'responds to []=' do + data = Squib::DataFrame.new basic + data[:a] = 2 + expect(data[:a]).to eq(2) + end + end + + context :columns do + it 'provides a list of columns' do + data = Squib::DataFrame.new basic + expect(data.columns).to eq %w(Name Cost) + end + end + + context :ncolumns do + it 'provides column count' do + data = Squib::DataFrame.new basic + expect(data.ncolumns).to eq 2 + end + end + + context :row do + it 'returns a hash of each row' do + data = Squib::DataFrame.new basic + expect(data.row(0)).to eq ({'Name' => 'Mage', 'Cost' => 1}) + expect(data.row(1)).to eq ({'Name' => 'Rogue', 'Cost' => 2}) + end + + it 'returns nil for uneven data' do + data = Squib::DataFrame.new uneven + expect(data.row(2)).to eq ({'Name' => 'Warrior', 'Cost' => nil}) + end + end + + context :nrows do + it 'returns a row count on even data' do + data = Squib::DataFrame.new basic + expect(data.nrows).to eq 3 + end + + it 'returns largest row count on uneven data' do + data = Squib::DataFrame.new basic + expect(data.nrows).to eq 3 + end + end + + context :json do + it 'returns quoty json' do + data = Squib::DataFrame.new basic + expect(data.to_json).to eq "{\"Name\":[\"Mage\",\"Rogue\",\"Warrior\"],\"Cost\":[1,2,3]}" + end + + it 'returns pretty json' do + data = Squib::DataFrame.new basic + str = <<-EOS +{ + "Name": [ + "Mage", + "Rogue", + "Warrior" + ], + "Cost": [ + 1, + 2, + 3 + ] +} +EOS + expect(data.to_pretty_json).to eq str.chomp + end + end + + context :def_column do + it 'creates name for Name column' do + data = Squib::DataFrame.new basic + expect(data).to respond_to(:name) + expect(data.name).to eq %w(Mage Rogue Warrior) + expect(data.cost).to eq [1, 2, 3] + end + + it 'does not redefine methods' do + data = Squib::DataFrame.new defined + expect(data).to respond_to(:nrows) + expect(data.nrows).to eq 3 + end + + it 'defines a new column from empty data frame' do + data = Squib::DataFrame.new + expect(data).not_to respond_to(:snark) + data['snark'] = [1,2,3] + expect(data.snark).to eq [1,2,3] + end + end + + context :snake_case do + subject { Squib::DataFrame.new } + it 'strips leading & trailing whitespace' do + expect(subject.send(:snake_case, ' A ')).to eq :a + end + + it 'converts space to _' do + expect(subject.send(:snake_case, 'A b')).to eq :a_b + end + + it 'handles multiwhitespace' do + expect(subject.send(:snake_case, 'A b')).to eq :a_b + end + + it 'handles camelcase' do + expect(subject.send(:snake_case, 'fooBar')).to eq :foo_bar + end + + it 'handles camelcase with multiple capitals' do + expect(subject.send(:snake_case, 'FOOBar')).to eq :foo_bar + end + end + + context :col? do + it 'returns true if a column exists' do + data = Squib::DataFrame.new basic + expect(data.col? 'Name').to be true + end + + it 'returns false if a column does not exist' do + data = Squib::DataFrame.new basic + expect(data.col? 'ROUS').to be false + end + end + + context :wrap_n_pad do + subject { Squib::DataFrame.new basic } + + it 'returns a handles a single line' do + expect(subject.send(:wrap_n_pad, 'a', 3)). + to eq "| a |\n" + end + + it 'wraps multiple a lines at 34 characters' do + expect(subject.send(:wrap_n_pad, 'a' * 40, 3)). + to eq <<-EOS +| aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | + | aaaaaa | +EOS + end + end + + context :to_pretty_text do + + let(:verbose) do + { + 'Name' => ['Mage', 'Rogue', 'Warrior'], + 'Cost' => [1, 2, 3], + 'Description' => [ + 'Magic, dude.', + 'I like to be sneaky', + 'I have a long story to tell to test the word-wrapping ability of pretty text formatting.' + ], + 'Snark' => [nil, '', ' '] + } + end + + it 'returns fun ascii art of the card' do + data = Squib::DataFrame.new verbose + expect(data.to_pretty_text).to eq <<-EOS + ╭------------------------------------╮ + Name | Mage | + Cost | 1 | +Description | Magic, dude. | + Snark | | + ╰------------------------------------╯ + ╭------------------------------------╮ + Name | Rogue | + Cost | 2 | +Description | I like to be sneaky | + Snark | | + ╰------------------------------------╯ + ╭------------------------------------╮ + Name | Warrior | + Cost | 3 | +Description | I have a long story to tell to tes | + | t the word-wrapping ability of pre | + | tty text formatting. | + Snark | | + ╰------------------------------------╯ +EOS + end + + let(:utf8_fun) do + { + 'Fun with UTF8!' => ['👊'], + } + end + + it 'is admittedly janky with multibyte chars' do + # I hate how Ruby handles multibyte chars. Blech! + data = Squib::DataFrame.new utf8_fun + expect(data.to_pretty_text).to eq <<-EOS + ╭------------------------------------╮ +Fun with UTF8! | 👊 | + ╰------------------------------------╯ +EOS + end + end + +end