Utility functions
=================

.. :doctest:

We're testing ``utils.py`` here:

    >>> from zest.releaser import utils
    >>> from pprint import pprint


Log level
---------

A ``-v`` on the commandline turns on debug level logging:

    >>> import sys
    >>> import logging
    >>> sys.argv[1:] = []
    >>> utils.parse_options()
    >>> utils.VERBOSE
    False
    >>> utils.loglevel() == logging.INFO
    True
    >>> sys.argv[1:] = ['-v']
    >>> utils.parse_options()
    >>> utils.VERBOSE
    True
    >>> utils.loglevel() == logging.DEBUG
    True
    >>> sys.argv[1:] = []


Version numbers
---------------

Strip all whitespace in a version:

    >>> utils.strip_version('1.0')
    '1.0'
    >>> utils.strip_version(' 1.0   dev  ')
    '1.0dev'

Remove development markers in various common forms:

    >>> utils.cleanup_version('1.0')
    '1.0'
    >>> utils.cleanup_version('1.0 dev')
    '1.0'
    >>> utils.cleanup_version('1.0 (svn/devel)')
    '1.0'
    >>> utils.cleanup_version('1.0 svn')
    '1.0'
    >>> utils.cleanup_version('1.0 devel 13')
    '1.0'
    >>> utils.cleanup_version('1.0 beta devel 13')
    '1.0 beta'
    >>> utils.cleanup_version('1.0.dev0')
    '1.0'
    >>> utils.cleanup_version('1.0.dev42')
    '1.0'


Asking input
------------

Asking input on the prompt is not unittestable unless we use the prepared
testing hack in utils.py:

    >>> utils.TESTMODE = True
    >>> utils.test_answer_book.set_answers([])

The default is True, so hitting enter (which means no input) returns True

    >>> utils.test_answer_book.set_answers([''])
    >>> utils.ask('Does mocking work?')
    Question: Does mocking work? (Y/n)?
    Our reply: <ENTER>
    True

A default of False also changes the Y/n to y/N:

    >>> utils.test_answer_book.set_answers([''])
    >>> utils.ask('Does mocking work?', default=False)
    Question: Does mocking work? (y/N)?
    Our reply: <ENTER>
    False

A default of None requires an answer:

    >>> utils.test_answer_book.set_answers(['', 'y'])
    >>> utils.ask('Does mocking work?', default=None)
    Question: Does mocking work? (y/n)?
    Our reply: <ENTER>
    Please explicitly answer y/n
    Question: Does mocking work? (y/n)?
    Our reply: y
    True

Y and n can be upper or lower case:

    >>> utils.test_answer_book.set_answers(['y'])
    >>> utils.ask('Does mocking work?', default=None)
    Question: Does mocking work? (y/n)?
    Our reply: y
    True
    >>> utils.test_answer_book.set_answers(['Y'])
    >>> utils.ask('Does mocking work?', default=None)
    Question: Does mocking work? (y/n)?
    Our reply: Y
    True
    >>> utils.test_answer_book.set_answers(['n'])
    >>> utils.ask('Does mocking work?', default=None)
    Question: Does mocking work? (y/n)?
    Our reply: n
    False
    >>> utils.test_answer_book.set_answers(['N'])
    >>> utils.ask('Does mocking work?', default=None)
    Question: Does mocking work? (y/n)?
    Our reply: N
    False

Yes and no are fine:

    >>> utils.test_answer_book.set_answers(['yes'])
    >>> utils.ask('Does mocking work?', default=None)
    Question: Does mocking work? (y/n)?
    Our reply: yes
    True
    >>> utils.test_answer_book.set_answers(['no'])
    >>> utils.ask('Does mocking work?', default=None)
    Question: Does mocking work? (y/n)?
    Our reply: no
    False

The y or n must be the first character, however, to prevent accidental input
from causing mishaps:

    >>> utils.test_answer_book.set_answers(['I reallY do not want it', 'n'])
    >>> utils.ask('Does mocking work?', default=None)
    Question: Does mocking work? (y/n)?
    Our reply: I reallY do not want it
    Please explicitly answer y/n
    Question: Does mocking work? (y/n)?
    Our reply: n
    False

You can also ask for a version number. Pass it as the default value:

    >>> utils.test_answer_book.set_answers([''])
    >>> utils.ask_version('New version', default='72.0')
    Question: New version [72.0]:
    Our reply: <ENTER>
    '72.0'
    >>> utils.test_answer_book.set_answers(['1.0', '', ''])
    >>> utils.ask_version('New version', default='72.0')
    Question: New version [72.0]:
    Our reply: 1.0
    '1.0'

Note that ``y`` or ``n`` are not accepted as answer for a version
number.  I see packages that get version ``y`` in postrelease because
someone quickly types ``y`` everywhere without looking at the
question.

    >>> utils.test_answer_book.set_answers(['Y', 'y', 'N', 'n', ''])
    >>> utils.ask_version('New version', default='72.0')
    Question: New version [72.0]:
    Our reply: Y
    y/n not accepted as version.
    Question: New version [72.0]:
    Our reply: y
    y/n not accepted as version.
    Question: New version [72.0]:
    Our reply: N
    y/n not accepted as version.
    Question: New version [72.0]:
    Our reply: n
    y/n not accepted as version.
    Question: New version [72.0]:
    Our reply: <ENTER>
    '72.0'


Not asking input
----------------

For running automatically, the ``--no-input`` option is available. By default
it is off:

    >>> utils.AUTO_RESPONSE
    False

We can switch it on:

    >>> utils.TESTMODE = False
    >>> utils.test_answer_book.set_answers([])
    >>> import sys
    >>> sys.argv[1:] = ['--no-input']
    >>> utils.parse_options()
    >>> utils.AUTO_RESPONSE
    True

This way, answers aren't even asked. With a default of yes:

    >>> utils.test_answer_book.set_answers([''])
    >>> utils.ask('Does mocking work?')
    True

A default of False:

    >>> utils.test_answer_book.set_answers([''])
    >>> utils.ask('Does mocking work?', default=False)
    False

A default of None requires an answer, which means we cannot run in
``--no-input`` mode. Return an error in this case:

    >>> utils.ask('Does mocking work?', default=None)
    Traceback (most recent call last):
    ...
    RuntimeError: The question 'Does...' requires a manual answer...

The default for versions is accepted, too:

    >>> utils.test_answer_book.set_answers([])
    >>> utils.ask('What mocking version, my liege?', default='72')
    '72'

Reset the defaults:

    >>> utils.TESTMODE = True
    >>> utils.AUTO_RESPONSE = False


Output filtering
----------------

Uploading to pypi returns quite some lines of output.  Only show the
first and last few lines, though we make sure we do not print too much
in case there are not 'enough' lines:

    >>> output = "a\nb\nc\nd\ne\nf\ng\n"
    >>> utils.show_interesting_lines(output)
    a
    b
    c
    d
    e
    f
    g
    <BLANKLINE>
    >>> import string
    >>> output = '\n'.join(string.ascii_lowercase)
    >>> utils.show_interesting_lines(output)
    Showing first few lines...
    a
    b
    c
    d
    e
    ...
    Showing last few lines...
    v
    w
    x
    y
    z

Just one line: no problem:

    >>> output = "just one line, no newlines"
    >>> utils.show_interesting_lines(output)
    just one line, no newlines

In case of errors, shown in red, we show all and ask what the user
wants to do.

    >>> output = "a\nb\nc\nd\ne\nf\ng\nh\ni\nj\nk\nl\nm\nn"
    >>> utils.show_interesting_lines(output)
    Showing first few lines...
    a
    b
    c
    d
    e
    ...
    Showing last few lines...
    j
    k
    l
    m
    n
    >>> from colorama import Fore
    >>> output = Fore.RED + output
    >>> utils.test_answer_book.set_answers([''])
    >>> utils.show_interesting_lines(output)
    Traceback (most recent call last):
    ...
    SystemExit: 1
    >>> utils.test_answer_book.set_answers(['y'])
    >>> utils.show_interesting_lines(output)
    RED a
    b
    c
    d
    e
    f
    g
    h
    i
    j
    k
    l
    m
    n
    Question: There were errors or warnings. Are you sure you want to continue? (y/N)?
    Our reply: y


Running commands
----------------

We can run commands.

    >>> print(utils.execute_command('echo "E.T. phone home."'))
    E.T. phone home.

We want to discover errors and show them in red.

    >>> Fore.RED
    '\x1b[31m'
    >>> Fore.MAGENTA
    '\x1b[35m'
    >>> utils.execute_command('ls some-non-existing-file')
    '\x1b[31mls...'

Warnings may also end up in the error output.  That may be unwanted.

    >>> warning = "warning: no previously-included files matching '*.pyc' found anywhere in distribution."
    >>> result = utils.execute_command('echo %s > /dev/stderr' % warning)
    >>> result.startswith(Fore.RED)
    False
    >>> result.startswith(Fore.MAGENTA)
    True
    >>> print(result)
    MAGENTA warning: no previously-included files matching *.pyc found anywhere in distribution.

One similar harmless warning by distutils does not get the 'warning:'
prefixed, so we handle it explicitly:

    >>> warning = "no previously-included directories found matching devsrc"
    >>> result = utils.execute_command('echo %s > /dev/stderr' % warning)
    >>> result.startswith(Fore.RED)
    False
    >>> result.startswith(Fore.MAGENTA)
    True
    >>> print(result)
    MAGENTA no previously-included directories found matching devsrc

Let's do a combination:

    >>> message = """
    ... Warn: What is the answer to life, the universe and everything?
    ...
    ... 41"""
    >>> result = utils.execute_command('echo "%s" > /dev/stderr' % message)
    >>> result
    '\x1b[35mWarn: What is the answer to life, the universe and everything?\n\n\x1b[31m41'
    >>> print(result)
    MAGENTA Warn: What is the answer to life, the universe and everything?
    <BLANKLINE>
    RED 41


Retrying commands
-----------------

Some commands may be retried.  For example, upload to PyPI may
temporarily fail.  Maybe the user has set a wrong password or username
in his .pypirc.  He can see the error, edit it (outside of the control
of zest.releaser) and retry the command.

Note that in these tests, the error output might not appear, because
the program is exited before the output is printed.  Or it may
appear twice, once printed and once as return value.

Also, the error output of the ``ls`` command we use here, can differ
significantly on different systems, so we do not check the exact line.

The user can choose to quit:

    >>> utils.test_answer_book.set_answers(['q'])
    >>> utils.execute_command('ls some-non-existing-file', allow_retry=True)
    Traceback (most recent call last):
    ...
    zest.releaser.utils.CommandException: Command failed: 'ls some-non-existing-file'

The user can choose to not retry and just continue:

    >>> utils.test_answer_book.set_answers(['n'])
    >>> result = utils.execute_command('ls some-non-existing-file', allow_retry=True)
    RED ls...
    RED There were errors or warnings.
    Question: Retry this command? [Yes/no/quit/?]
    Our reply: n
    >>> print(result)
    RED ls...

And there is the retry.  In the end you do have to choose something:

    >>> utils.test_answer_book.set_answers(['y', 'y', 'n'])
    >>> result = utils.execute_command('ls some-non-existing-file', allow_retry=True)
    RED ls...
    RED There were errors or warnings.
    Question: Retry this command? [Yes/no/quit/?]
    Our reply: y
    RED ls...
    RED There were errors or warnings.
    Question: Retry this command? [Yes/no/quit/?]
    Our reply: y
    RED ls...
    RED There were errors or warnings.
    Question: Retry this command? [Yes/no/quit/?]
    Our reply: n
    >>> print(result)
    RED ls...
    >>> utils.test_answer_book.set_answers(['y', 'y', 'q'])
    >>> utils.execute_command('ls some-non-existing-file', allow_retry=True)
    Traceback (most recent call last):
    ...
    zest.releaser.utils.CommandException: Command failed: 'ls some-non-existing-file'


Changelog header detection
--------------------------

Empty changelog:

    >>> lines = []
    >>> utils.extract_headings_from_history(lines)
    []

Various forms of version+date lines are recognised.  "unreleased" or a date in
paretheses:

    >>> lines = ["1.2 (unreleased)"]
    >>> pprint(utils.extract_headings_from_history(lines))
    [{'date': 'unreleased', 'line': 0, 'version': '1.2'}]
    >>> lines = ["1.1 (2008-12-25)"]
    >>> pprint(utils.extract_headings_from_history(lines))
    [{'date': '2008-12-25', 'line': 0, 'version': '1.1'}]

And dash-separated:

    >>> lines = ["1.0 - 1972-12-25"]
    >>> pprint(utils.extract_headings_from_history(lines))
    [{'date': '1972-12-25', 'line': 0, 'version': '1.0'}]

Versions with beta markers and spaces are fine:

    >>> lines = ["1.4 beta - unreleased"]
    >>> pprint(utils.extract_headings_from_history(lines))
    [{'date': 'unreleased', 'line': 0, 'version': '1.4 beta'}]

Multiple headers:

    >>> lines = ["1.2 (unreleased)",
    ...          "----------------",
    ...          "",
    ...          "- I did something.  [reinout]",
    ...          "",
    ...          "1.1 (2008-12-25)"
    ...          "----------------",
    ...          "",
    ...          "- Played Herodes in church play.  [reinout]",
    ...          ""]
    >>> pprint(utils.extract_headings_from_history(lines))
    [{'date': 'unreleased', 'line': 0, 'version': '1.2'},
     {'date': '2008-12-25', 'line': 5, 'version': '1.1'}]


reST headings
-------------

If a second line looks like a reST header line, fix up the length:

    >>> first = 'Hey, a potential heading'
    >>> second = '-------'
    >>> utils.fix_rst_heading(first, second)
    '------------------------'
    >>> second = '=='
    >>> utils.fix_rst_heading(first, second)
    '========================'
    >>> second = '``'
    >>> utils.fix_rst_heading(first, second)
    '````````````````````````'
    >>> second = '~~'
    >>> utils.fix_rst_heading(first, second)
    '~~~~~~~~~~~~~~~~~~~~~~~~'

No header line?  Just return the second line as-is:

    >>> second = 'just some text'
    >>> utils.fix_rst_heading(first, second)
    'just some text'

Empty line?  Just return it.

    >>> second = ''
    >>> utils.fix_rst_heading(first, second)
    ''

The second line must be uniform:

    >>> second = '- bullet point, no header'
    >>> utils.fix_rst_heading(first, second)
    '- bullet point, no header'


Safe setup.py running
---------------------

``setup_py()`` returns the ``python setup.py xyz`` command line by using
sys.executable.

    >>> cmd = utils.setup_py('cook a cow')
    >>> print(cmd.replace(sys.executable, 'python'))  # test normalization
    python setup.py cook a cow

When the setup.py arguments include arguments that indicate pypi interaction,
the python executable is replaced by ``echo`` for safety reasons.  This only
happens when TESTMODE is on:

    >>> utils.TESTMODE = True
    >>> print(utils.setup_py('upload'))
    echo MOCK setup.py upload
    >>> print(utils.setup_py('register'))
    echo MOCK setup.py register


Data dict documentation
-----------------------

The releasers have a data dict that is passed to entry points (and used
internally).  Because of the entry points, good documentation is necessary.
So we can check whether all keys have attached documentation:

    >>> data = {'commit_msg': 'get me some booze'}
    >>> documentation = {'commit_msg': 'Commit message for svn',
    ...                  'target': 'Some other thingy'}
    >>> utils.is_data_documented(data, documentation)
    Checking data dict

We print a warning when something is undocumented:

    >>> data = {'version': '0.1'}
    >>> utils.is_data_documented(data, documentation)
    Checking data dict
    Internal detail: key(s) ['version'] are not documented