Sun 의 공부 블로그

종합Newsweek과학NIST주식Money Morning

Google claimed quantum supremacy couple years back in 2019 and was later debunked by IBM.Building quantum systems is a feat of science and engineering and benchmarking them is a formidable challenge.through its association with white supremacy, evokes a repugnant political stance.outside of the rarified world of quantum computing experts that can put it in the appropriate context

There is a tendency to denigrate the poor.He never missed an opportunity to disparage his competitors.She was an incisive critic.The crude satire seems to be directed at the fashionable protest singers of the time.

They provide pragmatic and wise advice.Reverence means to respect and honour the Load as GOD.The devil goaded Jesus by quoting Psalm 91.to embark on a programme of unscrupulous manipulation.Deuteronomy.There is a nobility and dignity about all human life.Don't disqualify yourself.

The first sentence orients the reader by introducing the broader field in which the particular research is situated.This is typically done by recapitulating the results, discussing the limitations, and then revealing how the central contribution may catalyze future progress.We had to gloss over many finer points of writing, including word choice and grammar, the creativeprocess, and collaboratio..

Published papers that seem to refute your hypothesis or data are not etched in stone.The “For Authors” section of a website will have some nitty-gritty information.actors such as limitations of a methodology and other limits to generalizability (selection bias, unaddressed, or unappreciated confounders)A newcomer to the field should get a crash course in the field from this section.Essential dat..

목적S3에 저장해놓은 parquet 파일을 AWS ec2 등의 서버가 아닌 개인 컴퓨터에서 쉽게 읽고싶다.aws configure로 계정정보(access key, secret key)를 저장하는 것이 아닌 코드에서 계정정보를 관리하고싶다.필자는 pyarrow, s3fs가 동작하지 않아 빠르게 이용할 수 있는 다른 방법을 찾고 싶었다.코드import boto3import ioimport pandas as pd...s3_config = { "aws_access_key_id": "{ACCESS KEY}", "aws_secret_access_key": "{SECRET KEY}", "region_name": "{MY REGION}"}...def pd_read_s3_parquet(key, bucke..

목적 on-premise hdfs에서 s3로 데이터 올릴 때 get, put으로 데이터를 올리는 과정이 번거롭다 방법 hadoop distcp -Dfs.s3a.access.key=$AWS_ACCESS_KEY -Dfs.s3a.secret.key=$AWS_SECRET_KEY -Dfs.s3a.endpoint=$AWS_END_POINT $HDFS_SOURCE_PATH s3a://$S3_DEST_PATH fs.s3a.endpoint의 경우 AWS 서비스 엔드포인트를 참고 (ex. s3.ap-northeast-2.amazonaws.com)

목적 parquet의 경우, attribute name에 " ,;{}()\n\t=" 문자가 들어가면 rename을 시켜주어야한다. 에러메시지는 아래와 같다. 'Attribute name "my column" contains invalid character(s) among " ,;{}()\\n\\t=". Please use alias to rename it.;' ... pyspark.sql.utils.AnalysisException: 'Attribute name "some-ar ray" contains invalid character(s) among " ,;{}()\\n\\t=". Please use alias to rename it.;'​ json을 parquet로 저장 시 key값에 해당 문자들이 들어..

목적 json으로 저장된 파일을 partition을 추가하여 parquet로 저장하려한다. 이때, json 내에 빈 array가 있을 경우 inferSchema로 인해 array(string)의 스키마로 표현된다. 이 경우 다른 파일에서 array내에 struct 등의 다른 값이 들어올 경우 에러가 발생한다. pre-defined된 schema가 있다면 좋겠으나 필자의 경우는 그렇지 못하였다. 필자에게 발생한 에러메시지는 아래와 같다. java.lang.ClassCastException: optional binary element (UTF8) is not a group at org.apache.parquet.schema.Type.asGroupType(Type.java:207) at org.apache.s..