Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Yingjianw/additional safety rollout #602

Merged
merged 6 commits into from
Jul 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
import javax.ws.rs.PathParam;

import javax.ws.rs.core.MediaType;
import com.netflix.metacat.common.dto.notifications.ChildInfoDto;
import com.netflix.metacat.common.dto.ChildInfoDto;
import com.netflix.metacat.common.dto.ParentInfoDto;

import java.util.Set;

Expand Down Expand Up @@ -40,4 +41,24 @@ Set<ChildInfoDto> getChildren(
@PathParam("table-name")
String tableName
);

/**
* Return the list of parent.
* @param catalogName catalogName
* @param databaseName databaseName
* @param tableName tableName
* @return list of parent info dtos
*/
@GET
@Path("parents/catalog/{catalog-name}/database/{database-name}/table/{table-name}")
@Consumes(MediaType.APPLICATION_JSON)
@Produces(MediaType.APPLICATION_JSON)
Set<ParentInfoDto> getParents(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one the one hand, i would suggest we rename this to getParent since we are enforcing the one-parent-per-child rule and its a little confusing as the name implies we can expect more than one results. on the other hand, this API does technically return more than one result, so we would need to have a different return type to make this change. i have a weak preference to rename it and change the contract despite it becoming less flexible, but its up to you

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the api stays having multiple return values then plural is better so that the correspondence holds, otherwise singular. There is no current use case for multiple parents but keeping the flexibility is reasonable.

Copy link
Contributor Author

@stevie9868 stevie9868 Jul 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking the same thing before the implementation but ultimately choose to have the api flexibile that can return multiple parents.

Another benefit is if indeed somehow we end up with two parents on the same child table due to db isolation issues, we can still get them instead of just one.

@PathParam("catalog-name")
String catalogName,
@PathParam("database-name")
String databaseName,
@PathParam("table-name")
String tableName
);
}
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,8 @@
import com.netflix.metacat.common.dto.Sort;
import com.netflix.metacat.common.dto.StorageDto;
import com.netflix.metacat.common.dto.TableDto;
import com.netflix.metacat.common.dto.notifications.ChildInfoDto;
import com.netflix.metacat.common.dto.ChildInfoDto;
import com.netflix.metacat.common.dto.ParentInfoDto;
import com.netflix.metacat.common.server.connectors.ConnectorRequestContext;
import com.netflix.metacat.common.server.connectors.model.AuditInfo;
import com.netflix.metacat.common.server.connectors.model.CatalogInfo;
Expand All @@ -48,6 +49,7 @@
import com.netflix.metacat.common.server.connectors.model.StorageInfo;
import com.netflix.metacat.common.server.connectors.model.TableInfo;
import com.netflix.metacat.common.server.model.ChildInfo;
import com.netflix.metacat.common.server.model.ParentInfo;
import lombok.NonNull;
import org.dozer.CustomConverter;
import org.dozer.DozerBeanMapper;
Expand Down Expand Up @@ -287,4 +289,14 @@ public PartitionsSaveResponseDto toPartitionsSaveResponseDto(final PartitionsSav
public ChildInfoDto toChildInfoDto(final ChildInfo childInfo) {
return mapper.map(childInfo, ChildInfoDto.class);
}

/**
* Convert ParentInfo to ParentInfoDto.
*
* @param parentInfo parentInfo
* @return parentInfo dto
*/
public ParentInfoDto toParentInfoDto(final ParentInfo parentInfo) {
return mapper.map(parentInfo, ParentInfoDto.class);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -607,5 +607,40 @@ public interface Config {
* @return True if it should be.
*/
boolean shouldFetchOnlyMetadataLocationEnabled();

/**
* Whether we allow parent child relationship to be created.
*
* @return True if it should be.
*/
boolean isParentChildCreateEnabled();

/**
* Whether we allow renaming parent child relationship.
*
* @return True if it should be.
*/
boolean isParentChildRenameEnabled();

/**
* Whether we allow getting parent child relationship in the getTable call.
*
* @return True if it should be.
*/
boolean isParentChildGetEnabled();

/**
* Whether we allow dropping tables that are either parent or child.
*
* @return True if it should be.
*/
boolean isParentChildDropEnabled();

/**
* Get the parentChildRelationshipProperties config.
*
* @return parentChildRelationshipProperties
*/
ParentChildRelationshipProperties getParentChildRelationshipProperties();
}

Original file line number Diff line number Diff line change
Expand Up @@ -678,4 +678,29 @@ public boolean disablePartitionDefinitionMetadata() {
public boolean shouldFetchOnlyMetadataLocationEnabled() {
return this.metacatProperties.getHive().getIceberg().isShouldFetchOnlyMetadataLocationEnabled();
}

@Override
public boolean isParentChildCreateEnabled() {
return this.metacatProperties.getParentChildRelationshipProperties().isCreateEnabled();
}

@Override
public boolean isParentChildRenameEnabled() {
return this.metacatProperties.getParentChildRelationshipProperties().isRenameEnabled();
}

@Override
public boolean isParentChildGetEnabled() {
return this.metacatProperties.getParentChildRelationshipProperties().isGetEnabled();
}

@Override
public boolean isParentChildDropEnabled() {
return this.metacatProperties.getParentChildRelationshipProperties().isDropEnabled();
}

@Override
public ParentChildRelationshipProperties getParentChildRelationshipProperties() {
return this.metacatProperties.getParentChildRelationshipProperties();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
package com.netflix.metacat.common.server.properties;

import lombok.NonNull;
import org.springframework.core.env.Environment;

/**
* Entry point to entire property tree of Metacat namespace.
Expand All @@ -27,6 +28,8 @@
*/
@lombok.Data
public class MetacatProperties {
@NonNull
private Environment env;
@NonNull
private Data data = new Data();
@NonNull
Expand Down Expand Up @@ -67,4 +70,16 @@ public class MetacatProperties {
private AliasServiceProperties aliasServiceProperties = new AliasServiceProperties();
@NonNull
private RateLimiterProperties rateLimiterProperties = new RateLimiterProperties();
@NonNull
private ParentChildRelationshipProperties parentChildRelationshipProperties;

/**
* Constructor for MetacatProperties.
*
* @param env Spring Environment
*/
public MetacatProperties(final Environment env) {
this.env = env;
this.parentChildRelationshipProperties = new ParentChildRelationshipProperties(env);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
package com.netflix.metacat.common.server.properties;

import lombok.Data;
import lombok.extern.slf4j.Slf4j;
import org.springframework.core.env.Environment;

import javax.annotation.Nullable;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;
import java.util.stream.Collectors;

/**
* Parent Child Relationship service properties.
*
* @author yingjianw
*/
@Data
@Slf4j
public class ParentChildRelationshipProperties {
private static final String MAX_ALLOW_PER_TABLE_PER_REL_PROPERTY_NAME =
"metacat.parentChildRelationshipProperties.maxAllowPerTablePerRelConfig";
private static final String MAX_ALLOW_PER_DB_PER_REL_PROPERTY_NAME =
"metacat.parentChildRelationshipProperties.maxAllowPerDBPerRelConfig";
private static final String DEFAULT_MAX_ALLOW_PER_REL_PROPERTY_NAME =
"metacat.parentChildRelationshipProperties.defaultMaxAllowPerRelConfig";
private boolean createEnabled;
private boolean getEnabled;
private boolean renameEnabled;
private boolean dropEnabled;
private int maxAllow = 5;
private Map<String, Map<String, Integer>> maxAllowPerTablePerRelType = new HashMap<>();
private Map<String, Map<String, Integer>> maxAllowPerDBPerRelType = new HashMap<>();
private Map<String, Integer> defaultMaxAllowPerRelType = new HashMap<>();

/**
* Constructor.
*
* @param env Spring environment
*/
public ParentChildRelationshipProperties(@Nullable final Environment env) {
if (env != null) {
setMaxAllowPerTablePerRelType(
env.getProperty(MAX_ALLOW_PER_TABLE_PER_REL_PROPERTY_NAME, String.class, "")
);
setMaxAllowPerDBPerRelType(
env.getProperty(MAX_ALLOW_PER_DB_PER_REL_PROPERTY_NAME, String.class, "")
);
setDefaultMaxAllowPerRelType(
env.getProperty(DEFAULT_MAX_ALLOW_PER_REL_PROPERTY_NAME, String.class, "")
);
}
}

/**
* setMaxAllowPerTablePerRelType based on String config.
*
* @param configStr configString
*/
public void setMaxAllowPerTablePerRelType(@Nullable final String configStr) {
if (configStr == null || configStr.isEmpty()) {
return;
}
try {
this.maxAllowPerTablePerRelType = parseNestedConfigString(configStr);
} catch (Exception e) {
log.error("Fail to apply configStr = {} for maxAllowPerTablePerRelType", configStr, e);
}
}

/**
* setMaxAllowPerDBPerRelType based on String config.
*
* @param configStr configString
*/
public void setMaxAllowPerDBPerRelType(@Nullable final String configStr) {
if (configStr == null || configStr.isEmpty()) {
return;
}
try {
this.maxAllowPerDBPerRelType = parseNestedConfigString(configStr);
} catch (Exception e) {
log.error("Fail to apply configStr = {} for maxCloneAllowPerDBPerRelType", configStr);
}
}
/**
* setMaxCloneAllowPerDBPerRelType based on String config.
*
* @param configStr configString
*/
public void setDefaultMaxAllowPerRelType(@Nullable final String configStr) {
if (configStr == null || configStr.isEmpty()) {
return;
}
try {
this.defaultMaxAllowPerRelType =
Arrays.stream(configStr.split(";"))
.map(entry -> entry.split(","))
.collect(Collectors.toMap(
parts -> parts[0],
parts -> Integer.parseInt(parts[1])
));
} catch (Exception e) {
log.error("Fail to apply configStr = {} for defaultMaxAllowPerRelType", configStr);
}
}

private Map<String, Map<String, Integer>> parseNestedConfigString(final String configStr) {
return Arrays.stream(configStr.split(";"))
.map(entry -> entry.split(","))
.collect(Collectors.groupingBy(
parts -> parts[0],
Collectors.toMap(
parts -> parts[1],
parts -> Integer.parseInt(parts[2])
)
));
}
}
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
package com.netflix.metacat.common.server.usermetadata;
import com.netflix.metacat.common.QualifiedName;
import com.netflix.metacat.common.dto.notifications.ChildInfoDto;
import com.netflix.metacat.common.dto.ChildInfoDto;
import com.netflix.metacat.common.dto.ParentInfoDto;
import com.netflix.metacat.common.server.model.ChildInfo;
import com.netflix.metacat.common.server.model.ParentInfo;
import com.netflix.metacat.common.server.properties.ParentChildRelationshipProperties;

import java.util.Set;

Expand All @@ -24,13 +26,15 @@ public interface ParentChildRelMetadataService {
* @param childName the name of the child entity
* @param childUUID the uuid of the child
* @param relationType the type of the relationship
* @param prop properties config
*/
void createParentChildRelation(
QualifiedName parentName,
String parentUUID,
QualifiedName childName,
String childUUID,
String relationType
String relationType,
ParentChildRelationshipProperties prop
);

/**
Expand Down Expand Up @@ -98,9 +102,30 @@ Set<ChildInfo> getChildren(
/**
* get the set of children dto for the input name.
* @param name name
* @return a set of ChildInfo
* @return a set of ChildInfo dto
*/
Set<ChildInfoDto> getChildrenDto(
QualifiedName name
);

/**
* get the set of parent dto for the input name.
* @param name name
* @return a set of parentInfo dto
*/
Set<ParentInfoDto> getParentsDto(QualifiedName name);

/**
* return whether the table is a parent.
* @param tableName tableName
* @return true if it exists
*/
boolean isParentTable(final QualifiedName tableName);

/**
* return whether the table is a child.
* @param tableName tableName
* @return true if it exists
*/
boolean isChildTable(final QualifiedName tableName);
}
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
package com.netflix.metacat.common.dto.notifications;
package com.netflix.metacat.common.dto;

import com.netflix.metacat.common.dto.BaseDto;
import io.swagger.annotations.ApiModelProperty;
import lombok.AllArgsConstructor;
import lombok.Builder;
Expand All @@ -9,7 +8,7 @@
import lombok.NoArgsConstructor;

/**
* ChildInfo information.
* ChildInfo dto information.
*/
@SuppressWarnings("unused")
@Data
Expand Down
Loading
Loading