Course Notes

TEST YOUR UNDERSTANDING WITH CODE FROM THE COURSE

After watching the video, use the code that is supplied with the paid version of the course in order to test your understanding of logging in Python and Cloud Logging.

EXAMINE BASIC LOGGING CONFIGURATION LOCALLY AND ON THE CLOUD

Take a look at the following classes in tests/test_shared_configtools.py:

TestConfigureBasicLogging class

    def test_configure_basic_logging(self):
      self.assertEqual(len(logging.root.handlers), 0)
      configtools.configure_basic_logging()
      self.assertEqual(len(logging.root.handlers), 1)

                  

1. Run the function test_configure_basic_logging within the TestConfigureBasicLogging class. This code is designed to assign a basic date format, assign a basic structure to the log message, and test to see if some assignment has been made to the logger. This test passes if logging.root.handlers is equal to 1? Why should this test work?

  def test_configure_logging_local(self):
    fmt_string = '%(asctime)s.%(msecs)03d'
    fmt_string += ' | %(levelname)s'
    fmt_string += ' | %(name)s'
    fmt_string += ' | %(module)s'
    fmt_string += ' | %(funcName)s'
    fmt_string += ' | %(message)s'

    log_override = {'logging':
      {'console': 
        {'send': True,
         'level': 'INFO',
         'datefmt': '%Y-%m-%d %H:%M:%S',
         'format': fmt_string
        }
        , 'gcp': {'send': False}}}

    config = OmegaConf.merge(log_override)
    creds = None
    environ['LOG_NAME'] = 'test_logger'
    configtools.configure_logging(config.logging, creds)

    # This is the test of the actual logging
    log = logging.getLogger(environ['LOG_NAME'])
    log.info("This is a test")
    # log.warning("This is a warning")
    # log.error("This is an error")

    return
                  

2. Use a debugger to go line by line through the test_configure_logging_local function within the TestConfigureBasicLogging class. Enable the log.warning and log.error lines in this function. What difference do you see in the console after enabling these lines?

TestLoadCredentials class

    def test_load_credentials(self):
      environ['CREDENTIAL_SET'] = 'USER1'
      user_cred_set = \
        configtools.load_credentials('/yaml/credentials.yml')
      
      self.assertGreaterEqual(len(user_cred_set), 1)
      self.assertNotEqual(user_cred_set['google']['project_name'], 'machinelearningmastery-123456')

      return
                

3. The test_load_credentials function in the TestLoadCredentials class confirms that you have installed the libraries for OmegaConf correctly and that you can pull a configuration file into memory. As part of a previous lesson in the Machine Learning Mastery course, you were instructed to change all occurrences of the sample project (machinelearningmastery-123456) in the yaml/credentials.yml file to the project that you have in your Google Cloud Account. Run test_load_credentials and confirm that the test works. Why should this assertNotEqual test work?

TestConfigureCloudLogging class

class TestConfigureCloudLogging(unittest.TestCase):

  def test_configure_logging_gcp(self):
      ... 
      
      log = logging.getLogger(environ['LOG_NAME'])
      log.info("INFO - Simulated info")
      log.warning("WARNING - Simulated warning ")
      log.error("ERROR - Simulated error")

                    

4. Run test_configure_logging_gcp within test_shared_configtools. Make changes to the log.info line so that you can see warnings and errors within Google Cloud Logging.

USING THE YAML FORMAT FOR CONFIGURATION FILES

Take a look at yaml/config.yml that is part of the files that are included as part of the paid course. Among other things, this file gives configuration information about logging. Also, take a look at the OmegaConf documentation for examples of a YAML implementation.

5. OmegaConf allows the use of “dot notation” when referencing parameters in the YAML file. What string is found if config.yaml is loaded into memory as config and if config.logging.gcp.format is referenced?

Take a look at the load_configuration function in shared/configtools.py . This file is used in part to load parts of the config.yml into memory.

    def load_configuration(c_file, project_root):
      """
      Load full configuration data for all profiles
      found in a yaml file.
      
      :param c_file: Name of the configuration file.
      :param project_root: This is the root directory
       (e.g 'D:\\python\\machinelearningmastery')
      
       :return: Loaded file
      """
      c_file = f'{project_root}{c_file}'
      try:
          return OmegaConf.load(c_file)
      except FileNotFoundError:
          configure_basic_logging()
          logging.critical(f'Config file not found.')
          logging.critical('Exiting program.')
          raise SystemExit
      except Exception:
          configure_basic_logging()
          logging.critical(f'Error received', exc_info=True)
          logging.critical('Exiting program.')
          raise SystemExit
                

6. What are some reasons to have a FileNotFoundError when you are attempting to read a YAML file into memory?